class_06_neighborhood_methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "downtown-trading",
   "metadata": {},
   "source": [
    "# Collaborative filtering - neighborhood methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "verified-accommodation",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from IPython.display import Markdown, display, HTML\n",
    "from collections import defaultdict\n",
    "\n",
    "np.set_printoptions(edgeitems=10, linewidth=500)\n",
    "\n",
    "# Fix the dying kernel problem (only a problem in some installations - you can remove it, if it works without it)\n",
    "import os\n",
    "os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "educated-tourist",
   "metadata": {},
   "source": [
    "# Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "looking-feeling",
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_ratings_df = pd.read_csv(os.path.join(\"data\", \"movielens_small\", \"ratings.csv\")).rename(columns={'userId': 'user_id', 'movieId': 'item_id'})\n",
    "ml_movies_df = pd.read_csv(os.path.join(\"data\", \"movielens_small\", \"movies.csv\")).rename(columns={'movieId': 'item_id'})\n",
    "ml_df = pd.merge(ml_ratings_df, ml_movies_df, on='item_id')\n",
    "\n",
    "# Filter the data to reduce the number of movies\n",
    "left_ids = [1, 318, 1193, 1208, 1214, 1721, 2959, 3578, 4306, 109487]\n",
    "\n",
    "ml_ratings_df = ml_ratings_df.loc[ml_ratings_df['item_id'].isin(left_ids)]\n",
    "ml_movies_df = ml_movies_df.loc[ml_movies_df['item_id'].isin(left_ids)]\n",
    "ml_df = ml_df.loc[ml_df['item_id'].isin(left_ids)]\n",
    "\n",
    "display(ml_movies_df.head(10))\n",
    "\n",
    "print(\"Number of interactions left: {}\".format(len(ml_ratings_df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "plastic-brooklyn",
   "metadata": {},
   "source": [
    "# Shift item ids and user ids so that they are consecutive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "valuable-modem",
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df = ml_ratings_df.copy()\n",
    "\n",
    "unique_item_ids = interactions_df['item_id'].unique()\n",
    "item_id_mapping = dict(zip(unique_item_ids, list(range(len(unique_item_ids)))))\n",
    "item_id_reverse_mapping = dict(zip(list(range(len(unique_item_ids))), unique_item_ids))\n",
    "unique_user_ids = interactions_df['user_id'].unique()\n",
    "user_id_mapping = dict(zip(unique_user_ids, list(range(len(unique_user_ids)))))\n",
    "user_id_reverse_mapping = dict(zip(list(range(len(unique_user_ids))), unique_user_ids))\n",
    "\n",
    "interactions_df.replace({'item_id': item_id_mapping, 'user_id': user_id_mapping}, inplace=True)\n",
    "\n",
    "display(interactions_df.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "basic-meeting",
   "metadata": {},
   "source": [
    "# Get the number of items and users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "close-massachusetts",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_items = np.max(interactions_df['item_id']) + 1\n",
    "n_users = np.max(interactions_df['user_id']) + 1\n",
    "\n",
    "print(\"n_items={}\\nn_users={}\".format(n_items, n_users))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acknowledged-threshold",
   "metadata": {},
   "source": [
    "# Get the user-item interaction matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "extraordinary-mexico",
   "metadata": {},
   "outputs": [],
   "source": [
    "# mapping to int is necessary because of how iterrows works\n",
    "r = np.zeros(shape=(n_users, n_items))\n",
    "for idx, interaction in interactions_df.iterrows():\n",
    "    r[int(interaction['user_id'])][int(interaction['item_id'])] = 1\n",
    "    \n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lovely-password",
   "metadata": {},
   "source": [
    "# Calculate cosine similarities of users\n",
    "\n",
    "<center>\n",
    "$$\n",
    "    \\text{Sim}(\\vec{u}, \\vec{v}) = \\text{Cos}(\\vec{u}, \\vec{v}) = \\frac{\\vec{u} \\cdot \\vec{v}}{\\lVert u \\rVert \\lVert v \\rVert} = \\frac{\\sum_{i = 1}^n u_i v_i}{\\sqrt{\\sum_{i = 1}^n u_i^2} \\sqrt{\\sum_{i = 1}^n v_i^2}}\n",
    "$$\n",
    "</center>\n",
    "\n",
    "For interaction vectors cosine similarity changes from 0 to 1. 1 means that both vectors are identical. 0 means that they have no 1's in common."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-necklace",
   "metadata": {},
   "source": [
    "**Task 1.** Code the cosine method calculating the cosine similarity with above formula for two vectors (numpy arrays) $u$ and $v$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "rubber-detector",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosine(u, v):\n",
    "    # Write your code here\n",
    "\n",
    "print(cosine(np.array([1, 0, 1, 0]), np.array([1, 0, 0, 0])))\n",
    "print(cosine(np.array([1, 0, 1, 0]), np.array([1, 0, 1, 0])))\n",
    "print(cosine(np.array([1, 0, 1, 0]), np.array([0, 1, 0, 1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wound-biotechnology",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Cosine similarity between user 0 and 1\")\n",
    "user_id_1 = 0\n",
    "user_id_2 = 1\n",
    "print(r[user_id_1])\n",
    "print(r[user_id_2])\n",
    "print(cosine(r[user_id_1], r[user_id_2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "signal-orchestra",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Cosine similarity between user 0 and 5\")\n",
    "user_id_1 = 0\n",
    "user_id_2 = 5\n",
    "print(r[user_id_1])\n",
    "print(r[user_id_2])\n",
    "print(cosine(r[user_id_1], r[user_id_2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinguished-consequence",
   "metadata": {},
   "source": [
    "# Calculate Pearson similarities of users\n",
    "\n",
    "<center>\n",
    "$$\n",
    "    \\text{Sim}(\\vec{u}, \\vec{v}) = \\text{Pearson}(\\vec{u}, \\vec{v}) = \\frac{\\sum_{i = 1}^n (u_i - \\bar{u}) (v_i - \\bar{v})}{\\sqrt{\\sum_{i = 1}^n (u_i - \\bar{u})^2} \\sqrt{\\sum_{i = 1}^n (v_i - \\bar{v})^2}}\n",
    "$$\n",
    "</center>\n",
    "\n",
    "Correlation changes from -1 to 1. Correlation of 1 means that vectors are identical, -1 means they are opposites."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "present-smart",
   "metadata": {},
   "source": [
    "**Task 2.** Code the pearson method calculating the Peason similarity with the above formula for two vectors (numpy arrays) $u$ and $v$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pointed-deputy",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pearson(u, v):\n",
    "    # Write your code here\n",
    "\n",
    "print(pearson(np.array([1, 0, 1, 0]), np.array([1, 0, 0, 0])))\n",
    "print(pearson(np.array([1, 0, 1, 0]), np.array([1, 0, 1, 0])))\n",
    "print(pearson(np.array([1, 0, 1, 0]), np.array([0, 1, 0, 1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "numerous-assignment",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Pearson similarity between user 0 and 1\")\n",
    "user_id_1 = 0\n",
    "user_id_2 = 1\n",
    "print(r[user_id_1])\n",
    "print(r[user_id_2])\n",
    "print(pearson(r[user_id_1], r[user_id_2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "returning-porter",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Pearson similarity between user 0 and 5\")\n",
    "user_id_1 = 0\n",
    "user_id_2 = 5\n",
    "print(r[user_id_1])\n",
    "print(r[user_id_2])\n",
    "print(pearson(r[user_id_1], r[user_id_2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "social-cancer",
   "metadata": {},
   "source": [
    "# All cosine similarities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "national-tomato",
   "metadata": {},
   "source": [
    "**Task 3.** Calculate the entire matrix of cosine similarities between all users and print the first 15 rows and columns. Call the resulting matrix cos_sim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "conscious-symbol",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "\n",
    "print(\"Scalar products\")\n",
    "print(n_uv[:15, :15])\n",
    "print()\n",
    "\n",
    "print(\"Norms\")\n",
    "print(np.around(norms[:15], 3))\n",
    "print()\n",
    "\n",
    "print(\"Cosine similarities\")\n",
    "print(np.around(cos_sim[:15, :15], 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "silver-collect",
   "metadata": {},
   "source": [
    "# All Pearson similarities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alike-closer",
   "metadata": {},
   "source": [
    "**Task 4.** Calculate the entire matrix of Pearson similarities between all users and print the first 15 rows and columns. Call the resulting matrix pearson_sim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "yellow-franklin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "\n",
    "print(\"Scalar products\")\n",
    "print(np.around(n_uv[:15, :15], 3))\n",
    "print()\n",
    "\n",
    "print(\"Norms\")\n",
    "print(np.around(norms[:15], 3))\n",
    "print()\n",
    "\n",
    "print(\"Pearson similarities\")\n",
    "print(np.around(person_sim[:15, :15], 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lasting-porcelain",
   "metadata": {},
   "source": [
    "# Calculate scores of all items for user 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fifty-breeding",
   "metadata": {},
   "source": [
    "## Find n closest neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "several-mixer",
   "metadata": {},
   "source": [
    "**Task 5.** Generate a numpy array with n_neighbors ids of the closest neighbors of user 0 sorted decreasingly by the cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "advance-oriental",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.fill_diagonal(cos_sim, -1)\n",
    "\n",
    "user_id = 0\n",
    "n_neighbors = 10\n",
    "\n",
    "# Write your code here\n",
    "\n",
    "print(\"Nearest neighbors\")\n",
    "print(neighbor_ids)\n",
    "print()\n",
    "\n",
    "print(\"User {}\".format(user_id))\n",
    "print(r[user_id])\n",
    "print()\n",
    "for i in range(3):\n",
    "    print(\"User {}\".format(neighbor_ids[i]))\n",
    "    print(r[neighbor_ids[i]])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "described-marketplace",
   "metadata": {},
   "source": [
    "## Score all items\n",
    "\n",
    "<center>\n",
    "$$\n",
    "    \\text{score(i)} = \\frac{\\sum_{v \\in N(u)} \\text{Sim}(u, v) \\cdot v(i)}{\\sum_{v \\in N(u)} |\\text{Sim}(u, v)|}\n",
    "$$\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorporate-brake",
   "metadata": {},
   "source": [
    "**Task 6.** Code the score method to calculate the score of item $i$ for user $u$ based on a vector (numpy array) of similarities of this user to other users and an interaction vector (numpy array) of item $i$ with user's $u$ neighbors. Use the above formula. Print the score for user_id=0 and item_id=7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "complex-lawsuit",
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(similarities, v_i):\n",
    "    # Write your code here\n",
    "\n",
    "item_id = 7\n",
    "\n",
    "print(\"Interactions for nearest neighbors\")\n",
    "print(r[neighbor_ids])\n",
    "print()\n",
    "\n",
    "# Get similarities for the chosen user and his neighbors\n",
    "\n",
    "# Write your code here\n",
    "\n",
    "print(\"similarities\")\n",
    "print(similarities)\n",
    "print()\n",
    "\n",
    "# Get the interaction vector of the chosen item and the active user's neighbors\n",
    "\n",
    "# Write your code here\n",
    "\n",
    "print(\"v_i\")\n",
    "print(v_i)\n",
    "print()\n",
    "\n",
    "print(\"score for user_id={} and item_id={}\".format(user_id, item_id))\n",
    "print(score(similarities, v_i))\n",
    "\n",
    "\n",
    "# [1. 1. 1. 1. 1. 0. 0. 0. 0. 0.]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exclusive-processor",
   "metadata": {},
   "source": [
    "### Calculate and print scores for the first 10 items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesser-production",
   "metadata": {},
   "source": [
    "**Task 7.** Calculate and print scores for user_id=0 for the first 10 items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "governing-loading",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(r[user_id])\n",
    "\n",
    "for i in range(10):\n",
    "    # Write your code here\n",
    "    print(\"score for user_id={} and item_id={}\".format(user_id, i))\n",
    "    print(round(score(similarities, v_i), 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italian-helen",
   "metadata": {},
   "source": [
    "### The same scoring with a single operation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loose-candle",
   "metadata": {},
   "source": [
    "**Task 8.** Calculate scores for user_id=0 for the first 10 items in one operation using matrix multiplication. Print the resulting vector of scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sorted-horizontal",
   "metadata": {},
   "outputs": [],
   "source": [
    "item_ids = list(range(10))\n",
    "print(\"Neighbor ids\")\n",
    "print(neighbor_ids)\n",
    "print()\n",
    "print(\"Item ids\")\n",
    "print(item_ids)\n",
    "print()\n",
    "print(\"similarities\")\n",
    "print(similarities)\n",
    "print()\n",
    "\n",
    "# Get the interaction matrix of the chosen items and the active user's neighbors\n",
    "\n",
    "# Write your code here\n",
    "\n",
    "print(\"v_i\")\n",
    "print(v_i)\n",
    "print()\n",
    "\n",
    "# Calculate scores\n",
    "\n",
    "# Write your code here\n",
    "\n",
    "print(\"scores\")\n",
    "print(np.around(scores, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "endangered-stomach",
   "metadata": {},
   "source": [
    "# Load a bigger dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "prepared-fraction",
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_ratings_df = pd.read_csv(os.path.join(\"data\", \"movielens_small\", \"ratings.csv\")).rename(columns={'userId': 'user_id', 'movieId': 'item_id'})\n",
    "ml_movies_df = pd.read_csv(os.path.join(\"data\", \"movielens_small\", \"movies.csv\")).rename(columns={'movieId': 'item_id'})\n",
    "ml_df = pd.merge(ml_ratings_df, ml_movies_df, on='item_id')\n",
    "\n",
    "# Filter the data to reduce the number of movies\n",
    "seed = 6789\n",
    "rng = np.random.RandomState(seed=seed)\n",
    "left_ids = rng.choice(ml_movies_df['item_id'], size=100, replace=False)\n",
    "\n",
    "ml_ratings_df = ml_ratings_df.loc[ml_ratings_df['item_id'].isin(left_ids)]\n",
    "ml_movies_df = ml_movies_df.loc[ml_movies_df['item_id'].isin(left_ids)]\n",
    "ml_df = ml_df.loc[ml_df['item_id'].isin(left_ids)]\n",
    "\n",
    "display(ml_movies_df.head(10))\n",
    "\n",
    "print(\"Number of interactions left: {}\".format(len(ml_ratings_df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "needed-cyprus",
   "metadata": {},
   "source": [
    "**Task 9.** Fill in the code in the item-based version of the recommend method. Generate a numpy array of scores and a numpy array of chosen item ids (chosen_ids) with recommended item ids sorted by decreasing score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fancy-return",
   "metadata": {},
   "outputs": [],
   "source": [
    "from recommenders.recommender import Recommender\n",
    "\n",
    "class NearestNeighborsRecommender(Recommender):\n",
    "    \"\"\"\n",
    "    Nearest neighbors recommender allowing to do user-based or item-based collaborative filtering.\n",
    "\n",
    "    Possible similarity measures:\n",
    "        - 'cosine',\n",
    "        - 'pearson'.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.recommender_df = pd.DataFrame(columns=['user_id', 'item_id', 'score'])\n",
    "        self.interactions_df = None\n",
    "        self.item_id_mapping = None\n",
    "        self.user_id_mapping = None\n",
    "        self.item_id_reverse_mapping = None\n",
    "        self.user_id_reverse_mapping = None\n",
    "        self.r = None\n",
    "        self.similarities = None\n",
    "        self.most_popular_items = None\n",
    "\n",
    "        self.collaboration_type = 'user'\n",
    "        self.similarity_measure = 'cosine'\n",
    "        self.n_neighbors = 10\n",
    "        self.should_recommend_already_bought = False\n",
    "\n",
    "    def initialize(self, **params):\n",
    "        if 'n_neighbors' in params:\n",
    "            self.n_neighbors = params['n_neighbors']\n",
    "        if 'should_recommend_already_bought' in params:\n",
    "            self.should_recommend_already_bought = params['should_recommend_already_bought']\n",
    "\n",
    "    def fit(self, interactions_df, users_df, items_df):\n",
    "        \"\"\"\n",
    "        Training of the recommender.\n",
    "\n",
    "        :param pd.DataFrame interactions_df: DataFrame with recorded interactions between users and items\n",
    "            defined by user_id, item_id and features of the interaction.\n",
    "        :param pd.DataFrame users_df: DataFrame with users and their features defined by\n",
    "            user_id and the user feature columns.\n",
    "        :param pd.DataFrame items_df: DataFrame with items and their features defined\n",
    "            by item_id and the item feature columns.\n",
    "        \"\"\"\n",
    "\n",
    "        del users_df, items_df\n",
    "\n",
    "        # Shift item ids and user ids so that they are consecutive\n",
    "\n",
    "        unique_item_ids = interactions_df['item_id'].unique()\n",
    "        self.item_id_mapping = dict(zip(unique_item_ids, list(range(len(unique_item_ids)))))\n",
    "        self.item_id_reverse_mapping = dict(zip(list(range(len(unique_item_ids))), unique_item_ids))\n",
    "        unique_user_ids = interactions_df['user_id'].unique()\n",
    "        self.user_id_mapping = dict(zip(unique_user_ids, list(range(len(unique_user_ids)))))\n",
    "        self.user_id_reverse_mapping = dict(zip(list(range(len(unique_user_ids))), unique_user_ids))\n",
    "\n",
    "        interactions_df = interactions_df.copy()\n",
    "        interactions_df.replace({'item_id': self.item_id_mapping, 'user_id': self.user_id_mapping}, inplace=True)\n",
    "\n",
    "        # Get the number of items and users\n",
    "\n",
    "        self.interactions_df = interactions_df\n",
    "        n_items = np.max(interactions_df['item_id']) + 1\n",
    "        n_users = np.max(interactions_df['user_id']) + 1\n",
    "\n",
    "        # Get the user-item interaction matrix (mapping to int is necessary because of how iterrows works)\n",
    "        r = np.zeros(shape=(n_users, n_items))\n",
    "        for idx, interaction in interactions_df.iterrows():\n",
    "            r[int(interaction['user_id'])][int(interaction['item_id'])] = 1\n",
    "\n",
    "        if self.collaboration_type == 'item':\n",
    "            r = r.T\n",
    "\n",
    "        self.r = r\n",
    "\n",
    "        # Calculate all similarities\n",
    "\n",
    "        similarities = None\n",
    "        if self.similarity_measure == 'cosine':\n",
    "            n_uv = np.matmul(r, r.T)\n",
    "            norms = np.sqrt(np.diag(n_uv))\n",
    "            similarities = n_uv / norms[:, np.newaxis] / norms[np.newaxis, :]\n",
    "        elif self.similarity_measure == 'pearson':\n",
    "            r_shifted = r - np.mean(r, axis=1).reshape(-1, 1)\n",
    "            n_uv = np.matmul(r_shifted, r_shifted.T)\n",
    "            norms = np.sqrt(np.diag(n_uv))\n",
    "            norms[norms == 0] = 0.000001\n",
    "            similarities = n_uv / norms[:, np.newaxis] / norms[np.newaxis, :]\n",
    "\n",
    "        np.fill_diagonal(similarities, -1000)\n",
    "\n",
    "        self.similarities = similarities\n",
    "\n",
    "        # Find the most popular items for the cold start problem\n",
    "\n",
    "        offers_count = interactions_df.loc[:, ['item_id', 'user_id']].groupby(by='item_id').count()\n",
    "        offers_count = offers_count.sort_values('user_id', ascending=False)\n",
    "        self.most_popular_items = offers_count.index\n",
    "\n",
    "    def recommend(self, users_df, items_df, n_recommendations=1):\n",
    "        \"\"\"\n",
    "        Serving of recommendations. Scores items in items_df for each user in users_df and returns\n",
    "        top n_recommendations for each user.\n",
    "\n",
    "        :param pd.DataFrame users_df: DataFrame with users and their features for which\n",
    "            recommendations should be generated.\n",
    "        :param pd.DataFrame items_df: DataFrame with items and their features which should be scored.\n",
    "        :param int n_recommendations: Number of recommendations to be returned for each user.\n",
    "        :return: DataFrame with user_id, item_id and score as columns returning n_recommendations top recommendations\n",
    "            for each user.\n",
    "        :rtype: pd.DataFrame\n",
    "        \"\"\"\n",
    "\n",
    "        # Clean previous recommendations (iloc could be used alternatively)\n",
    "        self.recommender_df = self.recommender_df[:0]\n",
    "\n",
    "        # Handle users not in the training data\n",
    "\n",
    "        # Map item ids\n",
    "\n",
    "        items_df = items_df.copy()\n",
    "        items_df = items_df.loc[items_df['item_id'].isin(self.item_id_mapping)]\n",
    "        items_df.replace({'item_id': self.item_id_mapping}, inplace=True)\n",
    "\n",
    "        # Generate recommendations\n",
    "\n",
    "        for idx, user in users_df.iterrows():\n",
    "            recommendations = []\n",
    "\n",
    "            user_id = user['user_id']\n",
    "\n",
    "            if user_id in self.user_id_mapping:\n",
    "                chosen_ids = []\n",
    "                scores = []\n",
    "                mapped_user_id = self.user_id_mapping[user_id]\n",
    "\n",
    "                if self.collaboration_type == 'user':\n",
    "                    neighbor_ids = np.argsort(-self.similarities[mapped_user_id])[:self.n_neighbors]\n",
    "                    user_similarities = self.similarities[mapped_user_id][neighbor_ids]\n",
    "\n",
    "                    item_ids = items_df['item_id'].tolist()\n",
    "\n",
    "                    v_i = self.r[neighbor_ids][:, item_ids]\n",
    "\n",
    "                    scores = np.matmul(user_similarities, v_i) / np.maximum(np.sum(user_similarities), 0.0001)\n",
    "\n",
    "                    # Choose n recommendations based on highest scores\n",
    "                    if not self.should_recommend_already_bought:\n",
    "                        x_list = self.interactions_df.loc[\n",
    "                            self.interactions_df['user_id'] == mapped_user_id]['item_id'].tolist()\n",
    "                        scores[x_list] = -1e100\n",
    "\n",
    "                    chosen_ids = np.argsort(-scores)[:n_recommendations]\n",
    "\n",
    "                elif self.collaboration_type == 'item':\n",
    "                    \n",
    "                    # Write your code here\n",
    "\n",
    "                for item_id in chosen_ids:\n",
    "                    recommendations.append(\n",
    "                        {\n",
    "                            'user_id': self.user_id_reverse_mapping[mapped_user_id],\n",
    "                            'item_id': self.item_id_reverse_mapping[item_id],\n",
    "                            'score': scores[item_id]\n",
    "                        }\n",
    "                    )\n",
    "            else:  # For new users recommend most popular items\n",
    "                for i in range(n_recommendations):\n",
    "                    recommendations.append(\n",
    "                        {\n",
    "                            'user_id': user['user_id'],\n",
    "                            'item_id': self.item_id_reverse_mapping[self.most_popular_items[i]],\n",
    "                            'score': 1.0\n",
    "                        }\n",
    "                    )\n",
    "\n",
    "            user_recommendations = pd.DataFrame(recommendations)\n",
    "\n",
    "            self.recommender_df = pd.concat([self.recommender_df, user_recommendations])\n",
    "\n",
    "        return self.recommender_df\n",
    "    \n",
    "\n",
    "class UserBasedCosineNearestNeighborsRecommender(NearestNeighborsRecommender):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.collaboration_type = 'user'\n",
    "        self.similarity_measure = 'cosine'\n",
    "        \n",
    "        \n",
    "class UserBasedPearsonNearestNeighborsRecommender(NearestNeighborsRecommender):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.collaboration_type = 'user'\n",
    "        self.similarity_measure = 'pearson'\n",
    "        \n",
    "        \n",
    "class ItemBasedCosineNearestNeighborsRecommender(NearestNeighborsRecommender):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.collaboration_type = 'item'\n",
    "        self.similarity_measure = 'cosine'\n",
    "        \n",
    "\n",
    "class ItemBasedPearsonNearestNeighborsRecommender(NearestNeighborsRecommender):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.collaboration_type = 'item'\n",
    "        self.similarity_measure = 'pearson'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nonprofit-roads",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick test of the recommender\n",
    "\n",
    "nearest_neighbors_recommender = ItemBasedCosineNearestNeighborsRecommender()\n",
    "nearest_neighbors_recommender.initialize(n_neighbors=20)\n",
    "nearest_neighbors_recommender.fit(ml_ratings_df, None, ml_movies_df)\n",
    "recommendations = nearest_neighbors_recommender.recommend(pd.DataFrame([[1], [4], [6]], columns=['user_id']), ml_movies_df, 10)\n",
    "\n",
    "recommendations = pd.merge(recommendations, ml_movies_df, on='item_id', how='left')\n",
    "print(\"Recommendations\")\n",
    "display(recommendations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "framed-negative",
   "metadata": {},
   "source": [
    "# Training-test split evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "amended-future",
   "metadata": {},
   "outputs": [],
   "source": [
    "from evaluation_and_testing.testing import evaluate_train_test_split_implicit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "romantic-music",
   "metadata": {},
   "outputs": [],
   "source": [
    "ub_cos_nn_recommender = UserBasedCosineNearestNeighborsRecommender()\n",
    "ub_cos_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ub_cos_nn_tts_results = [['UserBasedCosineNearestNeighborsRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    ub_cos_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "ub_cos_nn_tts_results = pd.DataFrame(\n",
    "    ub_cos_nn_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ub_cos_nn_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sharp-hearts",
   "metadata": {},
   "outputs": [],
   "source": [
    "ub_pearson_nn_recommender = UserBasedPearsonNearestNeighborsRecommender()\n",
    "ub_pearson_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ub_pearson_nn_tts_results = [['UserBasedPearsonNearestNeighborsRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    ub_pearson_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "ub_pearson_nn_tts_results = pd.DataFrame(\n",
    "    ub_pearson_nn_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ub_pearson_nn_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mature-newton",
   "metadata": {},
   "outputs": [],
   "source": [
    "ib_cos_nn_recommender = ItemBasedCosineNearestNeighborsRecommender()\n",
    "ib_cos_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ib_cos_nn_tts_results = [['ItemBasedCosineNearestNeighborsRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    ib_cos_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "ib_cos_nn_tts_results = pd.DataFrame(\n",
    "    ib_cos_nn_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ib_cos_nn_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adjacent-crazy",
   "metadata": {},
   "outputs": [],
   "source": [
    "ib_pearson_nn_recommender = ItemBasedPearsonNearestNeighborsRecommender()\n",
    "ib_pearson_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ib_pearson_nn_tts_results = [['ItemBasedPearsonNearestNeighborsRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    ib_pearson_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "ib_pearson_nn_tts_results = pd.DataFrame(\n",
    "    ib_pearson_nn_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ib_pearson_nn_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "standing-tiffany",
   "metadata": {},
   "outputs": [],
   "source": [
    "from recommenders.amazon_recommender import AmazonRecommender\n",
    "\n",
    "amazon_recommender = AmazonRecommender()\n",
    "\n",
    "amazon_tts_results = [['AmazonRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    amazon_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "amazon_tts_results = pd.DataFrame(\n",
    "    amazon_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(amazon_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "saving-harrison",
   "metadata": {},
   "outputs": [],
   "source": [
    "from recommenders.tfidf_recommender import TFIDFRecommender\n",
    "\n",
    "tfidf_recommender = TFIDFRecommender()\n",
    "\n",
    "tfidf_tts_results = [['TFIDFRecommender'] + list(evaluate_train_test_split_implicit(\n",
    "    tfidf_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df))]\n",
    "\n",
    "tfidf_tts_results = pd.DataFrame(\n",
    "    tfidf_tts_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(tfidf_tts_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "random-source",
   "metadata": {},
   "outputs": [],
   "source": [
    "tts_results = pd.concat([ub_cos_nn_tts_results, ub_pearson_nn_tts_results, ib_cos_nn_tts_results, \n",
    "                         ib_pearson_nn_tts_results, amazon_tts_results, tfidf_tts_results]).reset_index(drop=True)\n",
    "display(tts_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "continued-harassment",
   "metadata": {},
   "source": [
    "# Leave-one-out evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exact-stuff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from evaluation_and_testing.testing import evaluate_leave_one_out_implicit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "prerequisite-lounge",
   "metadata": {},
   "outputs": [],
   "source": [
    "ub_cos_nn_recommender = UserBasedCosineNearestNeighborsRecommender()\n",
    "ub_cos_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ub_cos_nn_loo_results = [['UserBasedCosineNearestNeighborsRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    ub_cos_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "ub_cos_nn_loo_results = pd.DataFrame(\n",
    "    ub_cos_nn_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ub_cos_nn_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "olive-insurance",
   "metadata": {},
   "outputs": [],
   "source": [
    "ub_pearson_nn_recommender = UserBasedPearsonNearestNeighborsRecommender()\n",
    "ub_pearson_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ub_pearson_nn_loo_results = [['UserBasedPearsonNearestNeighborsRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    ub_pearson_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "ub_pearson_nn_loo_results = pd.DataFrame(\n",
    "    ub_pearson_nn_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ub_pearson_nn_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "determined-orleans",
   "metadata": {},
   "outputs": [],
   "source": [
    "ib_cos_nn_recommender = ItemBasedCosineNearestNeighborsRecommender()\n",
    "ib_cos_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ib_cos_nn_loo_results = [['ItemBasedCosineNearestNeighborsRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    ib_cos_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "ib_cos_nn_loo_results = pd.DataFrame(\n",
    "    ib_cos_nn_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ib_cos_nn_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "worse-management",
   "metadata": {},
   "outputs": [],
   "source": [
    "ib_pearson_nn_recommender = ItemBasedPearsonNearestNeighborsRecommender()\n",
    "ib_pearson_nn_recommender.initialize(n_neighbors=30)\n",
    "\n",
    "ib_pearson_nn_loo_results = [['ItemBasedPearsonNearestNeighborsRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    ib_pearson_nn_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "ib_pearson_nn_loo_results = pd.DataFrame(\n",
    "    ib_pearson_nn_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(ib_pearson_nn_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "social-escape",
   "metadata": {},
   "outputs": [],
   "source": [
    "from recommenders.amazon_recommender import AmazonRecommender\n",
    "\n",
    "amazon_recommender = AmazonRecommender()\n",
    "\n",
    "amazon_loo_results = [['AmazonRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    amazon_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "amazon_loo_results = pd.DataFrame(\n",
    "    amazon_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(amazon_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "behind-cambodia",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_recommender = TFIDFRecommender()\n",
    "\n",
    "tfidf_loo_results = [['TFIDFRecommender'] + list(evaluate_leave_one_out_implicit(\n",
    "    tfidf_recommender, ml_ratings_df.loc[:, ['user_id', 'item_id']], ml_movies_df, max_evals=300, seed=6789))]\n",
    "\n",
    "tfidf_loo_results = pd.DataFrame(\n",
    "    tfidf_loo_results, columns=['Recommender', 'HR@1', 'HR@3', 'HR@5', 'HR@10', 'NDCG@1', 'NDCG@3', 'NDCG@5', 'NDCG@10'])\n",
    "\n",
    "display(tfidf_loo_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "lightweight-password",
   "metadata": {},
   "outputs": [],
   "source": [
    "loo_results = pd.concat([ub_cos_nn_loo_results, ub_pearson_nn_loo_results, ib_cos_nn_loo_results, \n",
    "                         ib_pearson_nn_loo_results, amazon_loo_results, tfidf_loo_results]).reset_index(drop=True)\n",
    "display(loo_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "religious-valuable",
   "metadata": {},
   "source": [
    "**Task 10.** Add inverse of the euclidean distance as an eligible similarity measure in the nearest neighbors recommender and compare results of both the user- and item-based recommenders with this measure to other recommenders tested in this notebook. Create two new classes inheriting from the NearestNeighborsRecommender and name them:\n",
    "\n",
    "- UserBasedInvEuclideanNearestNeighborsRecommender,\n",
    "- ItemBasedInvEuclideanNearestNeighborsRecommender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "seventh-alert",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alien-island",
   "metadata": {},
   "source": [
    "**Task 11.** Find the optimal number of neighbors for the Item-Based Cosine Nearest Neighbors Recommender for $1 \\leq \\text{n_neighbors} \\leq 100$ and the train-test split testing scheme. Use seed=6789 and a set of interactions for a chosen subset of 100 movies. Use grid search to test all possibilities and compare it with the result of tuning with hyperopt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "annoying-consensus",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}