content-based-recommender-system-evaluation.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/gpreda/content-based-recommender-system-evaluation?scriptVersionId=128328186\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","id":"3fe7577c","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.006728,"end_time":"2023-05-04T17:35:36.375121","exception":false,"start_time":"2023-05-04T17:35:36.368393","status":"completed"},"tags":[]},"source":["\n","# Introduction\n","\n","This Notebook is evaluating Content Based Recommender System, by comparing it against random recommender. \n","\n","The content is as following:\n","\n","- Ingestion of data, using utility script for reading Movielens data (movie_lens_data).  \n","- Initialization of evaluator object, from utility script evaluator.   \n","- Definition of baseline random recommender.  \n","- Definition of Content Based Recommender System. It uses genres information to calculate the movies (items) similarity.\n","- Adds the baseline recomemnder, the naive top10 recommender and the Content Based Recommender System to the evaluator.  \n","- Perform evaluation, comparing the three algoritms.  \n","- Analyze the results.\n","\n"]},{"cell_type":"markdown","id":"94b20d00","metadata":{"papermill":{"duration":0.005084,"end_time":"2023-05-04T17:35:36.385828","exception":false,"start_time":"2023-05-04T17:35:36.380744","status":"completed"},"tags":[]},"source":["\n","# Analysis preparation"]},{"cell_type":"code","execution_count":1,"id":"81e14ef1","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:36.398394Z","iopub.status.busy":"2023-05-04T17:35:36.397999Z","iopub.status.idle":"2023-05-04T17:35:37.463481Z","shell.execute_reply":"2023-05-04T17:35:37.462268Z"},"papermill":{"duration":1.075258,"end_time":"2023-05-04T17:35:37.466522","exception":false,"start_time":"2023-05-04T17:35:36.391264","status":"completed"},"tags":[]},"outputs":[],"source":["import numpy as np\n","import pandas as pd\n","import re\n","import os\n","import heapq\n","from surprise import accuracy\n","from surprise import Dataset\n","from surprise import Reader\n","from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n","from surprise.model_selection import cross_validate\n","from surprise.model_selection import train_test_split\n","from surprise.model_selection import LeaveOneOut\n","from surprise import NormalPredictor, SVD"]},{"cell_type":"code","execution_count":2,"id":"f118d816","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:37.480174Z","iopub.status.busy":"2023-05-04T17:35:37.479442Z","iopub.status.idle":"2023-05-04T17:35:37.524926Z","shell.execute_reply":"2023-05-04T17:35:37.524021Z"},"papermill":{"duration":0.055475,"end_time":"2023-05-04T17:35:37.527626","exception":false,"start_time":"2023-05-04T17:35:37.472151","status":"completed"},"tags":[]},"outputs":[],"source":["from recommender_metrics import RecommenderMetrics\n","from movie_lens_data import MovieLensData\n","from evaluator import Evaluator"]},{"cell_type":"markdown","id":"b9860fbc","metadata":{"papermill":{"duration":0.005187,"end_time":"2023-05-04T17:35:37.538508","exception":false,"start_time":"2023-05-04T17:35:37.533321","status":"completed"},"tags":[]},"source":["# Read data\n","\n","We read the Movielens data using MovieLensData class from movie_lens_data utility script.\n","\n","First, we initialize the object."]},{"cell_type":"code","execution_count":3,"id":"0a528f45","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:37.551774Z","iopub.status.busy":"2023-05-04T17:35:37.551071Z","iopub.status.idle":"2023-05-04T17:35:37.557074Z","shell.execute_reply":"2023-05-04T17:35:37.556256Z"},"papermill":{"duration":0.015223,"end_time":"2023-05-04T17:35:37.559279","exception":false,"start_time":"2023-05-04T17:35:37.544056","status":"completed"},"tags":[]},"outputs":[],"source":["path = \"/kaggle/input/movielens-100k-dataset/ml-100k\"\n","movie_lens_data = MovieLensData(\n","    users_path = os.path.join(path, \"u.user\"),\n","    ratings_path = os.path.join(path, \"u.data\"), \n","    movies_path = os.path.join(path, \"u.item\"), \n","    genre_path = os.path.join(path, \"u.genre\") \n","    )"]},{"cell_type":"markdown","id":"17eb2cd5","metadata":{"papermill":{"duration":0.00512,"end_time":"2023-05-04T17:35:37.569932","exception":false,"start_time":"2023-05-04T17:35:37.564812","status":"completed"},"tags":[]},"source":["Then, we read:\n","- evaluation data   \n","- movie data  \n","- popularity rankings  "]},{"cell_type":"code","execution_count":4,"id":"0c85d7c7","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:37.582844Z","iopub.status.busy":"2023-05-04T17:35:37.582178Z","iopub.status.idle":"2023-05-04T17:35:42.638753Z","shell.execute_reply":"2023-05-04T17:35:42.637449Z"},"papermill":{"duration":5.066283,"end_time":"2023-05-04T17:35:42.641582","exception":false,"start_time":"2023-05-04T17:35:37.575299","status":"completed"},"tags":[]},"outputs":[],"source":["evaluation_data = movie_lens_data.read_ratings_data()\n","movie_data = movie_lens_data.read_movies_data()\n","popularity_rankings = movie_lens_data.get_popularity_ranks()"]},{"cell_type":"markdown","id":"e9a45e0c","metadata":{"papermill":{"duration":0.005289,"end_time":"2023-05-04T17:35:42.652858","exception":false,"start_time":"2023-05-04T17:35:42.647569","status":"completed"},"tags":[]},"source":["# Prepare evaluator\n","\n","\n","The evaluator will use the evaluation data to test the algorithms.\n","\n","Evaluator works with two components:  \n","- EvaluationData   \n","- EvaluatedAlgorithm  \n","\n","to define the data that will be used to test the algorithm(s) and the approach for testing the algorithm.  \n","The metrics to be used in testing are defined in RecommenderMetrics, from recomender_metrics utility script."]},{"cell_type":"code","execution_count":5,"id":"885f0f9a","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:42.667411Z","iopub.status.busy":"2023-05-04T17:35:42.666991Z","iopub.status.idle":"2023-05-04T17:35:48.314089Z","shell.execute_reply":"2023-05-04T17:35:48.31256Z"},"papermill":{"duration":5.658503,"end_time":"2023-05-04T17:35:48.317108","exception":false,"start_time":"2023-05-04T17:35:42.658605","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Number of full trainset users: 943\n","Number of full trainset items: 1682\n","Number of trainset users: 943\n","Number of trainset items: 1641\n","Size of testset: 25000\n","Estimating biases using als...\n","Computing the cosine similarity matrix...\n","Done computing similarity matrix.\n"]}],"source":["evaluator = Evaluator(evaluation_data, popularity_rankings)"]},{"cell_type":"markdown","id":"fe54c752","metadata":{"papermill":{"duration":0.005953,"end_time":"2023-05-04T17:35:48.330064","exception":false,"start_time":"2023-05-04T17:35:48.324111","status":"completed"},"tags":[]},"source":["# Add random recommender to evaluator\n","\n","We start by adding to the evaluator a random recommender system. The normal predictor will simply generate random recommendations."]},{"cell_type":"code","execution_count":6,"id":"2f5712fd","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.343775Z","iopub.status.busy":"2023-05-04T17:35:48.343327Z","iopub.status.idle":"2023-05-04T17:35:48.348944Z","shell.execute_reply":"2023-05-04T17:35:48.347701Z"},"papermill":{"duration":0.015098,"end_time":"2023-05-04T17:35:48.351245","exception":false,"start_time":"2023-05-04T17:35:48.336147","status":"completed"},"tags":[]},"outputs":[],"source":["algo_np = NormalPredictor()\n","evaluator.add_algorithm(algo_np, \"Random\")"]},{"cell_type":"markdown","id":"20677e9b","metadata":{"papermill":{"duration":0.005375,"end_time":"2023-05-04T17:35:48.362396","exception":false,"start_time":"2023-05-04T17:35:48.357021","status":"completed"},"tags":[]},"source":["# Prepare content based recommender system\n","\n","\n","Here we define our Content based recommender system.\n","\n","We define:\n","- init  \n","- fit  \n","- evaluate \n","\n","functions."]},{"cell_type":"code","execution_count":7,"id":"11aa5b2d","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.375599Z","iopub.status.busy":"2023-05-04T17:35:48.375176Z","iopub.status.idle":"2023-05-04T17:35:48.389273Z","shell.execute_reply":"2023-05-04T17:35:48.38799Z"},"papermill":{"duration":0.023667,"end_time":"2023-05-04T17:35:48.391683","exception":false,"start_time":"2023-05-04T17:35:48.368016","status":"completed"},"tags":[]},"outputs":[],"source":["from surprise import AlgoBase\n","from surprise import PredictionImpossible\n","\n","class ContentBasedRecSysKNN(AlgoBase):\n","\n","    def __init__(self, k=40, movie_data=None, sim_options={}):\n","        AlgoBase.__init__(self)\n","        self.k = k\n","        self.movie_data = movie_data\n","\n","    def fit(self, trainset):\n","        \"\"\"\n","        Fit function - calculate genre similarities\n","        Args\n","            trainset: the training set\n","        Returns\n","            self\n","        \"\"\"\n","        AlgoBase.fit(self, trainset)\n","        \n","        self.compute_genre_similarities()\n","    \n","        return self\n","    \n","    def estimate(self, u, i):\n","        \"\"\"\n","        Estimate function: evaluate the ratings for an user & item\n","        Process is:\n","            - gather the ratings for the user u\n","            - from the similarities (on items) between item i and the items\n","            from the ratings gathered (neighboring items)\n","            - order these neighbors \n","            - calculate the average simmilarities score weighted by user ratings\n","        Args\n","            u: current user\n","            i: current item\n","        Returns\n","            predicted ratings\n","        \"\"\"\n","        if not (self.trainset.knows_user(u) and self.trainset.knows_item(i)):\n","            raise PredictionImpossible('User and/or item is unkown.')\n","        \n","        # Build up similarity scores between this item and everything the user rated\n","        neighbors = []\n","        for rating in self.trainset.ur[u]:\n","            genre_similarity = self.genre_similarities[i,rating[0]]\n","            neighbors.append( (genre_similarity, rating[1]) )\n","        \n","        # Extract the top-K most-similar ratings (k is set at init)\n","        k_neighbors = heapq.nlargest(self.k, neighbors, key=lambda t: t[0])\n","        \n","        # Compute average sim score of K neighbors weighted by user ratings\n","        sim_total = weighted_sum = 0\n","        for (sim_score, rating) in k_neighbors:\n","            if (sim_score > 0):\n","                sim_total += sim_score\n","                weighted_sum += sim_score * rating\n","            \n","        if (sim_total == 0):\n","            raise PredictionImpossible('No neighbors')\n","\n","        predicted_rating = weighted_sum / sim_total\n","\n","        return predicted_rating\n","    \n","    def compute_genre_similarities(self):\n","        # compute similarities\n","        genre_columns = self.movie_data.columns[6:-1]\n","        genres = self.movie_data[genre_columns]\n","        # calculate cosine similarity using linear_kernel from sklearn\n","        self.genre_similarities = linear_kernel(genres, genres)\n","        return self"]},{"cell_type":"markdown","id":"2a6dcd63","metadata":{"papermill":{"duration":0.005327,"end_time":"2023-05-04T17:35:48.402704","exception":false,"start_time":"2023-05-04T17:35:48.397377","status":"completed"},"tags":[]},"source":["# Prepare naive top 10 RecSys"]},{"cell_type":"code","execution_count":8,"id":"ca808e76","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.41568Z","iopub.status.busy":"2023-05-04T17:35:48.415291Z","iopub.status.idle":"2023-05-04T17:35:48.42661Z","shell.execute_reply":"2023-05-04T17:35:48.425685Z"},"papermill":{"duration":0.020559,"end_time":"2023-05-04T17:35:48.42881","exception":false,"start_time":"2023-05-04T17:35:48.408251","status":"completed"},"tags":[]},"outputs":[],"source":["class Top10dRecSys(AlgoBase):\n","\n","    def __init__(self, ratings=None):\n","        AlgoBase.__init__(self)\n","        self.ratings = ratings\n","        \n","        self.items = []\n","\n","    def fit(self, trainset):\n","        \"\"\"\n","        Fit function - calculate top 10\n","        Args\n","            trainset: the training set\n","        Returns\n","            self\n","        \"\"\"\n","        AlgoBase.fit(self, trainset)\n","        \n","        self.prepare_rankings()\n","        return self\n","    \n","    def estimate(self, u, i):\n","        \"\"\"\n","        Estimate function: evaluate the ratings for an user & item\n","        Process is:\n","            - get the rating for item i (is not dependent of user) from \n","            the ranking dataframe\n","            - from the similarities (on items) between item i and the items\n","\n","        Args\n","            u: current user\n","            i: current item\n","        Returns\n","            predicted ratings\n","        \"\"\"\n","        \n","        if not (self.trainset.knows_user(u) and self.trainset.knows_item(i)):\n","            raise PredictionImpossible('User and/or item is unkown.')\n","\n","        # this is the rating for user u regarding item i\n","        # each item rating is the same, and is from self.rankings data, where\n","        # items rating is the average rating from all users per that item (if exists)\n","        predicted_rating = self.rankings.loc[self.rankings[\"item\"]==i][\"rating_score\"].values\n","        if predicted_rating:\n","            return predicted_rating[0]\n","        \n","        # if there is no rating for that item by any user, we just return averge rating/all\n","        return self.average_rating\n","    \n","    def prepare_rankings(self):\n","        # Collector\n","        self.rankings = pd.DataFrame(self.ratings.groupby('movie_id')['rating'].mean())\n","        self.rankings['rating_count'] = self.ratings.groupby('movie_id')['rating'].count()\n","        self.rankings = self.rankings.reset_index()\n","        self.rankings.columns = [\"item\", \"rating_score\", \"rating_count\"]\n","\n","        # Ranking\n","        self.rankings = self.rankings.sort_values(ascending=False,by=['rating_score', 'rating_count'])\n","        self.average_rating = self.rankings[\"rating_score\"].mean()\n","\n","    def get_top_n(self, top_n=10):\n","        # Serving\n","        return self.rankings.head(top_n)"]},{"cell_type":"markdown","id":"4a12c728","metadata":{"papermill":{"duration":0.005331,"end_time":"2023-05-04T17:35:48.439912","exception":false,"start_time":"2023-05-04T17:35:48.434581","status":"completed"},"tags":[]},"source":["# Add popularity ranking (naive top10) algo to evaluator"]},{"cell_type":"code","execution_count":9,"id":"371e9e78","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.453061Z","iopub.status.busy":"2023-05-04T17:35:48.452621Z","iopub.status.idle":"2023-05-04T17:35:48.45776Z","shell.execute_reply":"2023-05-04T17:35:48.45647Z"},"papermill":{"duration":0.01457,"end_time":"2023-05-04T17:35:48.460059","exception":false,"start_time":"2023-05-04T17:35:48.445489","status":"completed"},"tags":[]},"outputs":[],"source":["ratings = movie_lens_data.get_ratings()\n","top_10_rec_sys = Top10dRecSys(ratings)\n","evaluator.add_algorithm(top_10_rec_sys, \"Top10\")"]},{"cell_type":"markdown","id":"43b7704c","metadata":{"papermill":{"duration":0.005826,"end_time":"2023-05-04T17:35:48.471709","exception":false,"start_time":"2023-05-04T17:35:48.465883","status":"completed"},"tags":[]},"source":["# Add content based recommender system to evaluator"]},{"cell_type":"code","execution_count":10,"id":"f3101d82","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.485337Z","iopub.status.busy":"2023-05-04T17:35:48.484569Z","iopub.status.idle":"2023-05-04T17:35:48.489501Z","shell.execute_reply":"2023-05-04T17:35:48.488464Z"},"papermill":{"duration":0.01417,"end_time":"2023-05-04T17:35:48.491697","exception":false,"start_time":"2023-05-04T17:35:48.477527","status":"completed"},"tags":[]},"outputs":[],"source":["algo_content = ContentBasedRecSysKNN(movie_data=movie_data)\n","evaluator.add_algorithm(algo_content, \"ContentKNN\")"]},{"cell_type":"markdown","id":"7713d486","metadata":{"papermill":{"duration":0.005437,"end_time":"2023-05-04T17:35:48.502903","exception":false,"start_time":"2023-05-04T17:35:48.497466","status":"completed"},"tags":[]},"source":["# Evaluate the algorithms"]},{"cell_type":"markdown","id":"663aa88a","metadata":{"papermill":{"duration":0.005406,"end_time":"2023-05-04T17:35:48.514105","exception":false,"start_time":"2023-05-04T17:35:48.508699","status":"completed"},"tags":[]},"source":["After we initialized the evaluator, defined the algorithms to be tested and added them to the evaluator, we run now the evaluation."]},{"cell_type":"code","execution_count":11,"id":"9c54bdb0","metadata":{"execution":{"iopub.execute_input":"2023-05-04T17:35:48.527437Z","iopub.status.busy":"2023-05-04T17:35:48.526746Z","iopub.status.idle":"2023-05-04T17:36:04.607579Z","shell.execute_reply":"2023-05-04T17:36:04.606131Z"},"papermill":{"duration":16.090536,"end_time":"2023-05-04T17:36:04.610305","exception":false,"start_time":"2023-05-04T17:35:48.519769","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Evaluating  Random ...\n","Evaluating accuracy...\n","Analysis complete.\n","Evaluating  Top10 ...\n","Evaluating accuracy...\n"]},{"name":"stderr","output_type":"stream","text":["/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:44: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n"]},{"name":"stdout","output_type":"stream","text":["Analysis complete.\n","Evaluating  ContentKNN ...\n","Evaluating accuracy...\n","Analysis complete.\n","\n","\n","Algorithm  RMSE       MAE       \n","Random     1.5258     1.2250    \n","Top10      1.2698     1.0215    \n","ContentKNN 1.0735     0.8536    \n","\n","Legend:\n","\n","RMSE:      Root Mean Squared Error. Lower values mean better accuracy.\n","MAE:       Mean Absolute Error. Lower values mean better accuracy.\n"]}],"source":["evaluator.evaluate(do_top_n=False)"]},{"cell_type":"markdown","id":"ec00d2df","metadata":{"papermill":{"duration":0.00582,"end_time":"2023-05-04T17:36:04.62268","exception":false,"start_time":"2023-05-04T17:36:04.61686","status":"completed"},"tags":[]},"source":["# Analysis of results\n","\n","\n","We can see that the content based recommender system gives better (smaller) results than the Top10 and the random recommender, for both metrics used."]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.12"},"papermill":{"default_parameters":{},"duration":40.644465,"end_time":"2023-05-04T17:36:05.753166","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-05-04T17:35:25.108701","version":"2.4.0"}},"nbformat":4,"nbformat_minor":5}