src/experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "File: experiments.ipynb <br>\n",
    "Author: Jan Koci <br>\n",
    "Date: 05/2019 <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial of performed experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ipython notebook shows all experiments performed in the course of this thesis. It tries to give a brief description of the implemented models, their usage and the process of their evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all we will import modules that contain useful functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import data_utils, helpers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to work with the recommender models, we have to first load the dataset containing user-item interactions. The dataset that will be used here is stored in the '/data/processed/' directory. To load it we can simply use the __load_df_pickle__ function from __data_utils__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(264468, 4)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = data_utils.load_df_pickle('../data/processed/dataset2.pkl')\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>page_url</th>\n",
       "      <th>page_name</th>\n",
       "      <th>time</th>\n",
       "      <th>uid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://developers.redhat.com/blog/2017/10/04/...</td>\n",
       "      <td>rhd|blog|2017|10|4|red-hat-adds-go-clangllvm-r...</td>\n",
       "      <td>579</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://developers.redhat.com/blog/2017/11/01/...</td>\n",
       "      <td>rhd|blog|2017|11|1|getting-started-llvm-toolset</td>\n",
       "      <td>2038</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://developers.redhat.com/blog/2018/07/07/...</td>\n",
       "      <td>rhd|blog|2018|7|7|yum-install-gcc7-clang</td>\n",
       "      <td>1650</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://developers.redhat.com/blog/2017/10/04/...</td>\n",
       "      <td>rhd|blog|2017|10|4|red-hat-updates-python-php-...</td>\n",
       "      <td>1033</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://developers.redhat.com/blog/2015/06/01/...</td>\n",
       "      <td>rhd|blog|2015|6|1|five-different-ways-handle-l...</td>\n",
       "      <td>433</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            page_url  \\\n",
       "0  https://developers.redhat.com/blog/2017/10/04/...   \n",
       "1  https://developers.redhat.com/blog/2017/11/01/...   \n",
       "2  https://developers.redhat.com/blog/2018/07/07/...   \n",
       "3  https://developers.redhat.com/blog/2017/10/04/...   \n",
       "4  https://developers.redhat.com/blog/2015/06/01/...   \n",
       "\n",
       "                                           page_name  time  uid  \n",
       "0  rhd|blog|2017|10|4|red-hat-adds-go-clangllvm-r...   579    0  \n",
       "1    rhd|blog|2017|11|1|getting-started-llvm-toolset  2038    0  \n",
       "2           rhd|blog|2018|7|7|yum-install-gcc7-clang  1650    0  \n",
       "3  rhd|blog|2017|10|4|red-hat-updates-python-php-...  1033    1  \n",
       "4  rhd|blog|2015|6|1|five-different-ways-handle-l...   433    2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the dataset consists of interactions that contain the following values:\n",
    "- page_url: URL of the page\n",
    "- page_name: string containing info about the page\n",
    "- time: time in seconds the user spent on the page\n",
    "- uid: user identifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will also load separate sets for training testing and validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = data_utils.load_df_pickle('../data/processed/train_df.pkl')\n",
    "test_df = data_utils.load_df_pickle('../data/processed/test_df.pkl')\n",
    "validation_df = data_utils.load_df_pickle('../data/processed/validation_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_df = (250663, 4)\n",
      "test_df = (13805, 4)\n",
      "validation_df = (17658, 4)\n"
     ]
    }
   ],
   "source": [
    "print(\"train_df = {0}\\ntest_df = {1}\\nvalidation_df = {2}\".format(train_df.shape, test_df.shape, validation_df.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. SVD recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first model that will be shown is the model using __singular value decomposition__ (SVD). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model is implemented in file __svd_model.py__ as a class called __SVDModel__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from svd_model import SVDModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd_recommender = SVDModel(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The whole dataset is passed during initialization. The model will use it to only create user and item mappings. It will not be used to train the model. <br>\n",
    "To evaluate this model we did not use the whole __df__ dataset, as performing the SVD consumes a lot of CPU. Therefore we used a reduced version of the dataset, containing only users that interacted with at least 3 different pages. This dataset is called __df_3__. We also need to create new training and testing sets from the reduced dataset. For this the __train_test_df__ function can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_3 = data_utils.load_df_pickle('../data/processed/df_3.pkl')\n",
    "train_df3, test_df3 = helpers.train_test_df(df_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df_3 = (69353, 4)\n",
      "train_df3 = (55548, 4)\n",
      "test_df3 = (13805, 4)\n"
     ]
    }
   ],
   "source": [
    "print(\"df_3 = {0}\\ntrain_df3 = {1}\\ntest_df3 = {2}\".format(df_3.shape, train_df3.shape, test_df3.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that we are good to go and train the model. We will train it using the default values, that means it will use the binary metric for computing the confidence of interactions. Bare in mind this command will take a while to execute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##################### SVD model #####################\n",
      "[1]  Making sparse interaction matrix\n",
      "[2]  Performing SVD on interaction matrix\n",
      "[DONE]  SVD successfull\n"
     ]
    }
   ],
   "source": [
    "svd_recommender.train(train_df3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the model is trained we can simply call the __predict__ and __recommend__ functions to predict a score a user would give an item or create a list of top n recommendations for a user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.4861567944205734e-16"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd_recommender.predict(uid=19, url='https://developers.redhat.com/blog/2016/04/26/fedora-media-writer-the-fastest-way-to-create-live-usb-boot-media')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('https://developers.redhat.com/blog/2018/03/13/eclipse-vertx-first-application',\n",
       "  0.02870943859519231),\n",
       " ('https://developers.redhat.com/blog/2016/10/10/business-process-management-in-a-microservices-world',\n",
       "  0.023943767806097478),\n",
       " ('https://developers.redhat.com/blog/2013/01/21/welcome-to-the-red-hat-developer-blog',\n",
       "  0.021294564339902657),\n",
       " ('https://developers.redhat.com/blog/2013/08/01/php-5-4-on-rhel-6-using-rhscl',\n",
       "  0.02102011260995658),\n",
       " ('https://developers.redhat.com/blog/2018/08/22/reducing-data-inconsistencies-with-red-hat-process-automation-manager',\n",
       "  0.02079838576723706)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd_recommender.recommend(uid=19, n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. ALS recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now have a look at another created model. This model is an example of a traditional collaborative filtering technique using matrix factorization. In particular it uses the __Alternating Least Squares__ (ALS) method to decompose the interaction matrix into two matrices, one containing the user factors and the other item factors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To construct this model we need to import the __ImplicitALS__ class implemented in __als_model.py__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from als_model import ImplicitALS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "als_recommender = ImplicitALS(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train the model using the log confidence metric. That means we need to provide the train method with required values of the model's hyperparameters. For that we use the __optimal_parameters.py__ file, that contains the optimal parameters of our models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimal_parameters import als_rank_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the model can be trained on the whole dataset. Therefore we will use the __train_df__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##################### ALS model #####################\n",
      "[1]  Creating sparse interaction matrix\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 1.5/76 [00:00<00:06, 12.35it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[2]  Fitting the matrix to the model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 76.0/76 [00:05<00:00, 13.51it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[DONE]  ALS successfull\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "als_recommender.train(train_df, **als_rank_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the training we can once again call the __predict__ and __recommend__ methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available',\n",
       "  0.99473566),\n",
       " ('https://developers.redhat.com/blog/2018/03/19/sso-made-easy-keycloak-rhsso',\n",
       "  0.8615376),\n",
       " ('https://developers.redhat.com/blog/2018/05/07/announcing-amq-streams-apache-kafka-on-openshift',\n",
       "  0.8342068),\n",
       " ('https://developers.redhat.com/blog/2016/04/26/fedora-media-writer-the-fastest-way-to-create-live-usb-boot-media',\n",
       "  0.78171504),\n",
       " ('https://developers.redhat.com/videos/youtube/hy9aVrTNufQ', 0.7493656)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "als_recommender.recommend(uid=19, n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Doc2Vec recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on to another model, this time it an example of a conten-based model. Content-based models create recommendations based on the content of their items. Our model works with the __Doc2Vec__ method to create document vectors of our articles using their __metadata__. First let's have a look at what metadata we are talking about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = data_utils.load_metadata_json('../data/json/metadata_07_03_blogs.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': ['Kubernetes: Your Next Application Server'],\n",
       " 'type': ['blogpost'],\n",
       " 'tags': ['Containers',\n",
       "  'devnation',\n",
       "  'feed_group_name_nonmiddleware',\n",
       "  'feed_name_redhat_developer_blog',\n",
       "  'Java',\n",
       "  'Kubernetes',\n",
       "  'microservices'],\n",
       " 'date': ['2018-12-07T12:56:02.000Z'],\n",
       " 'description': ['Watch Burr Sutter in this week’s DevNation change how you think about application servers in today’s world of containers. Get the slides:\\xa0bit.ly/kubeappserver In the Java ecosystem, we have historically been enamored with the concept of the “application server,” the runtime engine that not only gave us portable APIs such as JMS, JAX-RS, JSF, and EJB but also gave us critical runtime infrastructure...']}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata['https://developers.redhat.com/blog/2018/12/07/kubernetes-application-server']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metadata contain the following information:\n",
    "- __title__ of the page\n",
    "- its __type__ (can be webpage, blogpost or video)\n",
    "- __tags__ that were assigned to it\n",
    "- the __date__ it was published\n",
    "- a __description__ containing the first paragraph of its actual content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The __Doc2Vec__ recommender takes the __title__ and the __description__ of each item and concatenates them together. It then uses the concatenated text to create its document representation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do so it first needs to transform the text representing an item into an object of the __TaggedDocument__ class, defined in the __gensim__ library. This object will store each item as a tuple, where the second value will be an integer serving as an identifier of the item and the first value a list containing the original text split into words. To simplify this process we created a class called __Doc2VecInput__, that takes in the dictionary containing metadata of all items and transforms it into a list of __TaggedDocument__ objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from doc2vec_class import Doc2VecInput"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_input = Doc2VecInput(metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_input.fit_transform()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TaggedDocument(words=['openshift', '4.0', 'developer', 'preview', 'on', 'aws', 'is', 'up', 'and', 'running', 'the', 'openshift', '4.0', 'developer', 'preview', 'is', 'available', 'for', 'amazon', 'web', 'services', '(', 'aws', ')', ',', 'and', 'if', 'you', '’', 're', 'anything', 'like', 'me', ',', 'you', 'want', 'to', 'be', 'among', 'the', 'first', 'to', 'get', 'your', 'hands', 'on', 'it', '.', 'the', 'starting', 'point', 'is', 'try.openshift.com', ',', 'where', 'you', '’', 'll', 'find', 'overview', 'information', 'and', 'that', 'important', '“', 'get', 'started', '”', 'button', '.', 'click', 'it', 'and', 'you', '’', 're', 'off', 'to', 'the', 'big', 'show', '.', '(', 'if', 'you', 'aren', '’', 't', 'a', 'red', 'hat', 'developer', 'member', ',', 'this', 'is', 'your', 'reason', 'to', 'sign'], tags=[0])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_input.input[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommender itself is implemented in class __Doc2VecModel__. One problem that occurs here, is that we do not have the metadata of every single page from the dataset. Similarly there are some items that are in the metadata dictionary but are not in the dataset. To deal with this problem we created a class called __RecommenderDataFrame__. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from data_utils import RecommenderDataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct an object of this class and pass it the item mappings created by the __Doc2VecInput__ class. This class then filters all interactions to contain only items that occur in the __url_2_id__ mappings created from the metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original = (264468, 4)\n",
      "filtered = (229229, 4)\n"
     ]
    }
   ],
   "source": [
    "dataframe = RecommenderDataFrame(df, url_2_id=doc_input.url_2_id)\n",
    "print('original = {0}\\nfiltered = {1}'.format(df.shape, dataframe.df.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This whole process is, however, hidden inside the __Doc2VecModel__ class. Therefore we only pass it our data set and the __Doc2VecInput__ object and the class will filter the interactions itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from doc2vec_model import Doc2VecModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc2vec_recommender = Doc2VecModel(train_df, doc_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the recommender is ready to create the document vectors from our items. To do so we can choose from two possible options. We can either train the __Doc2Vec__ model locally, only on our articles, or we can use a pre-trained model, created on English Wikipedia pages. That means we can choose between two methods:\n",
    "- __train__: to train the model locally\n",
    "- __load_pretrained__: to use the pre-trained model\n",
    "\n",
    "In this example we will show the load_pretrained method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc2vec_recommender.load_pretrained()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc2vec_recommender.doc_vectors[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc2vec_recommender.user_vectors[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see this model creates 300-dimensional document and user vectors. The __load_pretrained__ method provides only the binary confidence metric and therefore the user vectors are computed as a sum of their item vectors. After that we can once again call the __predict__ and __recommend__ methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('https://developers.redhat.com/blog/2018/06/04/red-hat-fuse-7-is-now-available',\n",
       "  21.578247927886967),\n",
       " ('https://developers.redhat.com/blog/2018/02/27/red-hat-jboss-fuse-7-tech-preview',\n",
       "  21.48103429673055),\n",
       " ('https://developers.redhat.com/videos/youtube/g5xeKuPo8Uw',\n",
       "  20.90573511479384),\n",
       " ('https://developers.redhat.com/videos/vimeo/33538130', 20.391794040539455),\n",
       " ('https://developers.redhat.com/videos/youtube/a0DXIspd1Zs',\n",
       "  20.333339194239716)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc2vec_recommender.recommend(uid=19, n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to use the document vectors to find the most similar items using the __nearest_items__ method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('https://developers.redhat.com/blog/2018/05/31/introducing-the-kafka-cdi-library',\n",
       "  9.623635029993466),\n",
       " ('https://developers.redhat.com/videos/youtube/QYbXDp4Vu-8',\n",
       "  8.260617221657947),\n",
       " ('https://developers.redhat.com/videos/youtube/mcbdnMDERX0',\n",
       "  8.13403926267652),\n",
       " ('https://developers.redhat.com/videos/vimeo/44390131', 7.807490549702573),\n",
       " ('https://developers.redhat.com/videos/youtube/F2lYSF25-Ek',\n",
       "  7.744083812481144),\n",
       " ('https://developers.redhat.com/videos/youtube/a0DXIspd1Zs',\n",
       "  7.694184206997353),\n",
       " ('https://developers.redhat.com/videos/youtube/6ZwL2sgKR3w',\n",
       "  7.671404735998803),\n",
       " ('https://developers.redhat.com/videos/youtube/x3QCrb6zCKA',\n",
       "  7.642057166797322)]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nearest = doc2vec_recommender.nearest_items('https://developers.redhat.com/blog/2018/05/31/introducing-the-kafka-cdi-library', 8)\n",
    "nearest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have a look at the titles of the nearest items. That shoud be more informative than their URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Introducing the Kafka-CDI Library']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata['https://developers.redhat.com/blog/2018/05/31/introducing-the-kafka-cdi-library']['title']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Introducing the Kafka-CDI Library'],\n",
       " ['Kafka and Debezium | DevNation Live'],\n",
       " ['Reactive systems with Eclipse Vert.x and Red Hat OpenShift'],\n",
       " ['Getting Started with EAP6 on OpenShift using JBoss Developer Studio'],\n",
       " ['An open platform to support digital transformation'],\n",
       " ['Developing cloud-ready Camel microservice'],\n",
       " ['2012 Red Hat Summit: Tuning & Benchmarking JBoss Enterprise Application Platform Session Clustering'],\n",
       " ['Kafka Streams for Event Driven Microservices | DevNation Live']]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[metadata[url]['title'] for url, _ in nearest]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare it to te nearest items found by the __ALS__ recommender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Announcing AMQ Streams: Apache Kafka on OpenShift'],\n",
       " ['Why Kubernetes Is the New Application Server'],\n",
       " ['How to run Kafka on Openshift, the enterprise Kubernetes, with AMQ Streams'],\n",
       " ['Getting started with OpenShift Java S2I'],\n",
       " ['Deploying a Spring Boot App with MySQL on OpenShift'],\n",
       " ['Welcome Apache Kafka to the Kubernetes Era!'],\n",
       " ['Patterns for distributed transactions within a microservices architecture'],\n",
       " ['Configuring Spring Boot on Kubernetes with ConfigMap']]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[metadata[url]['title'] for url, _ in als_recommender.nearest_items('https://developers.redhat.com/blog/2018/05/31/introducing-the-kafka-cdi-library', 8)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. SkipGram recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last implemented model is inspired by the Skip-gram method with negative sampling. This method was originally used to create word vectors in the __Word2Vec__ method. We tried to apply this approach for the recommendation problem to see if it can deal with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is implemented in the __SkipGramRecommender__ class. To use the model we have to prepare some data. First we need document vectors of our articles. For this we can use the vectors created with __Doc2VecModel__. We also need a dataframe object that tells the recommender, what tags are assigned to each item. We already prepared this dataframe and saved it into the __url_tags.pkl__ file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_tags = data_utils.load_df_pickle('../data/processed/url_tags.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://developers.redhat.com/blog/2019/03/07/...</td>\n",
       "      <td>amazon web services</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://developers.redhat.com/blog/2019/03/07/...</td>\n",
       "      <td>aws</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://developers.redhat.com/blog/2019/03/07/...</td>\n",
       "      <td>cloud</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://developers.redhat.com/blog/2019/03/07/...</td>\n",
       "      <td>containers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://developers.redhat.com/blog/2019/03/07/...</td>\n",
       "      <td>developer preview</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 url                  tag\n",
       "0  https://developers.redhat.com/blog/2019/03/07/...  amazon web services\n",
       "1  https://developers.redhat.com/blog/2019/03/07/...                  aws\n",
       "2  https://developers.redhat.com/blog/2019/03/07/...                cloud\n",
       "3  https://developers.redhat.com/blog/2019/03/07/...           containers\n",
       "4  https://developers.redhat.com/blog/2019/03/07/...    developer preview"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The __url_tags__ contains an item url and one tag assigned to the item in each row. We then pass this dataframe together with the document vectors to the __SkipGram__ recommender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skip_gram_recommender import SkipGramRecommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The __SkipGram__ recommender needs to receive a __RecommenderDataFrame__ object. Once again we will use only the reduced __df_3__ dataset, as the __SkipGram__ recommender is very memory consuming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_df3 = (55548, 4)\n",
      "train_dataframe3 = (47859, 4)\n"
     ]
    }
   ],
   "source": [
    "train_dataframe3 = data_utils.RecommenderDataFrame(train_df3, url_2_id=doc2vec_recommender.url_2_id)\n",
    "print(\"train_df3 = {0}\\ntrain_dataframe3 = {1}\".format(train_df3.shape, train_dataframe3.df.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "skip_gram_recommender = SkipGramRecommender(train_dataframe3, doc2vec_recommender.doc_vectors, url_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will load a pretrained model using the __load_model__ method. This model was originaly trained on _Google Colaboratory_ using a _Tesla T4_ GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimal_parameters import skip_gram_optimum\n",
    "skip_gram_recommender.load_model('../data/processed/skip_gram_learned.pt', **skip_gram_optimum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has now mapped all the parameters to the CPU. After that we can call its __predict__ and __recommend__ methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('https://developers.redhat.com/videos/youtube/8TX1OGHd1M0', 8.239341),\n",
       " ('https://developers.redhat.com/videos/youtube/8mlXKzBF2qA', 6.3137283),\n",
       " ('https://developers.redhat.com/videos/youtube/WYUsuHHDn2w', 5.905406),\n",
       " ('https://developers.redhat.com/videos/youtube/Z08FEd2r458', 5.6563663),\n",
       " ('https://developers.redhat.com/videos/youtube/Zdlyhhm-DdE', 4.857724)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skip_gram_recommender.recommend(19, n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have all recommenders ready, we can compare them using the __Evaluator__ class. This class provides three evaluation metrics that can be used for this task: \n",
    "- RANK evaluation metric\n",
    "- Recall at k\n",
    "- Precision at k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usage of this class is very simple. One only passes the recommender to its init method and calls one of the evaluation metrics with a dataset that will be used for the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from evaluation import Evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(als_recommender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.907955282425101"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.rank_evaluation(validation_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.16737376577257218"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.recall_at_k(validation_df, k=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.02641586309927941"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.precision_at_k(validation_df, k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the __Doc2Vec__ and __SkipGram__ recommenders we need to create __RecommenderDataFrame__ objects from the test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataframe = data_utils.RecommenderDataFrame(test_df, url_2_id=doc2vec_recommender.url_2_id, uid_2_id=doc2vec_recommender.uid_2_id)\n",
    "test_dataframe3 = data_utils.RecommenderDataFrame(test_df3, url_2_id=skip_gram_recommender.url_2_id, uid_2_id=skip_gram_recommender.uid_2_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now evaluate all our models and show all results in a table. First we will evaluate them using the __test_df__ dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(als_recommender)\n",
    "als_rank = evaluator.rank_evaluation(test_df)\n",
    "als_recall = evaluator.recall_at_k(test_df, k=10)\n",
    "als_precision = evaluator.precision_at_k(test_df, k=10)\n",
    "\n",
    "evaluator = Evaluator(svd_recommender)\n",
    "svd_rank = evaluator.rank_evaluation(test_df3)\n",
    "svd_recall = evaluator.recall_at_k(test_df3, k=10)\n",
    "svd_precision = evaluator.precision_at_k(test_df3, k=10)\n",
    "\n",
    "evaluator = Evaluator(doc2vec_recommender)\n",
    "d2v_rank = evaluator.rank_evaluation(test_dataframe.df)\n",
    "d2v_recall = evaluator.recall_at_k(test_dataframe.df, k=10)\n",
    "d2v_precision = evaluator.precision_at_k(test_dataframe.df, k=10)\n",
    "\n",
    "evaluator = Evaluator(skip_gram_recommender)\n",
    "sg_rank = evaluator.rank_evaluation(test_dataframe3.df)\n",
    "sg_recall = evaluator.recall_at_k(test_dataframe3.df, k=10)\n",
    "sg_precision = evaluator.precision_at_k(test_dataframe3.df, k=10)\n",
    "\n",
    "results = [('ALS', als_rank, als_recall, als_precision), \n",
    "           ('SVD', svd_rank, svd_recall, svd_precision),\n",
    "           ('Doc2Vec', d2v_rank, d2v_recall, d2v_precision),\n",
    "           ('SkipGram', sg_rank, sg_recall, sg_precision)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TEST SET EVALUATION RESULTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>Recommender</th><th>RANK</th><th>Recall</th><th>Precision</th></tr><tr><td>ALS</td><td>11.17</td><td>0.1316</td><td>0.0564</td></tr><tr><td>SVD</td><td>49.91</td><td>0.0073</td><td>0.0035</td></tr><tr><td>Doc2Vec</td><td>35.88</td><td>0.0219</td><td>0.0080</td></tr><tr><td>SkipGram</td><td>37.66</td><td>0.0034</td><td>0.0014</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import HTML, display\n",
    "html = \"<table><tr><th>Recommender</th><th>RANK</th><th>Recall</th><th>Precision</th></tr>\"\n",
    "for model,rank,recall,prec in results:\n",
    "    html += \"<tr><td>{0}</td><td>{1:.2f}</td><td>{2:.4f}</td><td>{3:.4f}</td></tr>\".format(model, rank, recall, prec)\n",
    "html += '</table>'\n",
    "display(HTML(html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will evaluate all recommenders on the __validation_df__. Note that we have to transform it to __RecommenderDataFrame__ class to be able to use it with our __Doc2Vec__ and __SkipGram__ recommenders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "validation_df = (17658, 4)\n",
      "validation_dataframe = (10500, 4)\n"
     ]
    }
   ],
   "source": [
    "validation_dataframe = data_utils.RecommenderDataFrame(validation_df, url_2_id=doc2vec_recommender.url_2_id, uid_2_id=doc2vec_recommender.uid_2_id)\n",
    "print(\"validation_df = {0}\\nvalidation_dataframe = {1}\".format(validation_df.shape, validation_dataframe.df.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "validation_df = (17658, 4)\n",
      "validation_dataframe_sg = (5050, 4)\n"
     ]
    }
   ],
   "source": [
    "validation_dataframe_sg = data_utils.RecommenderDataFrame(validation_df, url_2_id=skip_gram_recommender.url_2_id, uid_2_id=skip_gram_recommender.uid_2_id)\n",
    "print(\"validation_df = {0}\\nvalidation_dataframe_sg = {1}\".format(validation_df.shape, validation_dataframe_sg.df.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(als_recommender)\n",
    "als_rank = evaluator.rank_evaluation(validation_df)\n",
    "als_recall = evaluator.recall_at_k(validation_df, k=10)\n",
    "als_precision = evaluator.precision_at_k(validation_df, k=10)\n",
    "\n",
    "evaluator = Evaluator(svd_recommender)\n",
    "svd_rank = evaluator.rank_evaluation(validation_df)\n",
    "svd_recall = evaluator.recall_at_k(validation_df, k=10)\n",
    "svd_precision = evaluator.precision_at_k(validation_df, k=10)\n",
    "\n",
    "evaluator = Evaluator(doc2vec_recommender)\n",
    "d2v_rank = evaluator.rank_evaluation(validation_dataframe.df)\n",
    "d2v_recall = evaluator.recall_at_k(validation_dataframe.df, k=10)\n",
    "d2v_precision = evaluator.precision_at_k(validation_dataframe.df, k=10)\n",
    "\n",
    "evaluator = Evaluator(skip_gram_recommender)\n",
    "sg_rank = evaluator.rank_evaluation(validation_dataframe_sg.df)\n",
    "sg_recall = evaluator.recall_at_k(validation_dataframe_sg.df, k=10)\n",
    "sg_precision = evaluator.precision_at_k(validation_dataframe_sg.df, k=10)\n",
    "\n",
    "results_validation = [('ALS', als_rank, als_recall, als_precision), \n",
    "                      ('SVD', svd_rank, svd_recall, svd_precision),\n",
    "                      ('Doc2Vec', d2v_rank, d2v_recall, d2v_precision),\n",
    "                      ('SkipGram', sg_rank, sg_recall, sg_precision)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VALIDATION SET EVALUATION RESULTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>Recommender</th><th>RANK</th><th>Recall</th><th>Precision</th></tr><tr><td>ALS</td><td>13.91</td><td>0.1674</td><td>0.0264</td></tr><tr><td>SVD</td><td>42.70</td><td>0.0482</td><td>0.0071</td></tr><tr><td>Doc2Vec</td><td>31.89</td><td>0.1512</td><td>0.0202</td></tr><tr><td>SkipGram</td><td>36.31</td><td>0.0150</td><td>0.0028</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html = \"<table><tr><th>Recommender</th><th>RANK</th><th>Recall</th><th>Precision</th></tr>\"\n",
    "for model,rank,recall,prec in results_validation:\n",
    "    html += \"<tr><td>{0}</td><td>{1:.2f}</td><td>{2:.4f}</td><td>{3:.4f}</td></tr>\".format(model, rank, recall, prec)\n",
    "html += '</table>'\n",
    "display(HTML(html))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}