recsys_rl.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WBbEvG1aixZo"
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "import os\n",
    "import pickle\n",
    "import random\n",
    "import requests\n",
    "import time\n",
    "import tqdm\n",
    "\n",
    "from IPython.core.debugger import set_trace\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pytorch_ranger import Ranger\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F \n",
    "import torch.utils.data as td\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "\n",
    "from utils import (EvalDataset, OUNoise, Prioritized_Buffer, get_beta, \n",
    "                   preprocess_data, to_np, hit_metric, dcg_metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k7YiOW9TixZs"
   },
   "outputs": [],
   "source": [
    "data_dir = \"data\"\n",
    "rating = \"ml-1m.train.rating\"\n",
    "\n",
    "params = {\n",
    "    'batch_size': 512,\n",
    "    'embedding_dim': 8,\n",
    "    'hidden_dim': 16,\n",
    "    'N': 5, # memory size for state_repr\n",
    "    'ou_noise':False,\n",
    "    \n",
    "    'value_lr': 1e-5,\n",
    "    'value_decay': 1e-4,\n",
    "    'policy_lr': 1e-5,\n",
    "    'policy_decay': 1e-6,\n",
    "    'state_repr_lr': 1e-5,\n",
    "    'state_repr_decay': 1e-3,\n",
    "    'log_dir': 'logs/final/',\n",
    "    'gamma': 0.8,\n",
    "    'min_value': -10,\n",
    "    'max_value': 10,\n",
    "    'soft_tau': 1e-3,\n",
    "    \n",
    "    'buffer_size': 1000000\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Problem statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditional recommendation task can be treated as sequental desicion making problem.\n",
    "Recommender (i.e. agent) interacts with users (i.e. environment) to sequentally suggest set of items.\n",
    "The goal is to maximize clients' satisfaction (i.e. reward).\n",
    "More specifically:\n",
    "- State is a vector $a \\in R^{3\\cdot embedding\\_dim}$ computed using the user embedding and the embeddings of `N` latest positive interactions. In the code (replay buffer) state is represented  by `(user, memory)`\n",
    "- Action is a vector $a \\in R^{embedding\\_dim}$. To get ranking score we took dot product of\n",
    "the action and the item embedding (similar to word2vec and other embedding models).\n",
    "- Reward is taken from user-item matrix (1 if rating > 3, 0 otherwise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reinforcement Learning can help recommendation at least in 2 ways.\n",
    "1. User’s preference on previous items will affect his choice on the next items. \n",
    "User tends to give a higher rating if he has consecutively received more satisfied items (and vice versa). \n",
    "So, it would be more reasonable to model the recommendation as a sequential decision making process.\n",
    "2. It is important to use long-term planning in recommendations. For example, after reading the weather forecast, the user is not willing\n",
    "to read similar news. On the other hand, after watching funny videos or reading memes the user can constanly do the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "qabQkULCixZv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skip loading data/ml-1m.train.rating\n"
     ]
    }
   ],
   "source": [
    "# Movielens (1M) data from the https://github.com/hexiangnan/neural_collaborative_filtering\n",
    "if not os.path.isdir('./data'):\n",
    "    os.mkdir('./data')\n",
    "    \n",
    "file_path = os.path.join(data_dir, rating)\n",
    "if os.path.exists(file_path):\n",
    "    print(\"Skip loading \" + file_path)\n",
    "else:\n",
    "    with open(file_path, \"wb\") as tf:\n",
    "        print(\"Load \" + file_path)\n",
    "        r = requests.get(\"https://raw.githubusercontent.com/hexiangnan/neural_collaborative_filtering/master/Data/\" + rating)\n",
    "        tf.write(r.content)\n",
    "        \n",
    "(train_data, train_matrix, test_data, test_matrix, \n",
    " user_num, item_num, appropriate_users) = preprocess_data(data_dir, rating)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Observation space**. As mentioned before, to get state we need `N` latest positive items (`memory`) and embedding of user. `State_Repr_Module` transform it to the vector of dimensionality `embedding_dim * 3`.\n",
    "\n",
    "- **Action space**. For every user we sample nonrelated items (the same count as related). All `available_items` which wasn't viewed before form action space.\n",
    "\n",
    "Given a state we get action embedding, compute dot product between this embedding and embeddings of all items in action space, take 1 top ranked item, compute reward, update `viewed_items` and memory, and store transition in buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class Env():\n",
    "    def __init__(self, user_item_matrix):\n",
    "        self.matrix = user_item_matrix\n",
    "        self.item_count = item_num\n",
    "        self.memory = np.ones([user_num, params['N']]) * item_num\n",
    "        # memory is initialized as [item_num] * N for each user\n",
    "        # it is padding indexes in state_repr and will result in zero embeddings\n",
    "\n",
    "    def reset(self, user_id):\n",
    "        self.user_id = user_id\n",
    "        self.viewed_items = []\n",
    "        self.related_items = np.argwhere(self.matrix[self.user_id] > 0)[:, 1]\n",
    "        self.num_rele = len(self.related_items)\n",
    "        self.nonrelated_items = np.random.choice(\n",
    "            list(set(range(self.item_count)) - set(self.related_items)), self.num_rele)\n",
    "        self.available_items = np.zeros(self.num_rele * 2)\n",
    "        self.available_items[::2] = self.related_items\n",
    "        self.available_items[1::2] = self.nonrelated_items\n",
    "        \n",
    "        return torch.tensor([self.user_id]), torch.tensor(self.memory[[self.user_id], :])\n",
    "    \n",
    "    def step(self, action, action_emb=None, buffer=None):\n",
    "        initial_user = self.user_id\n",
    "        initial_memory = self.memory[[initial_user], :]\n",
    "        \n",
    "        reward = float(to_np(action)[0] in self.related_items)\n",
    "        self.viewed_items.append(to_np(action)[0])\n",
    "        if reward:\n",
    "            if len(action) == 1:\n",
    "                self.memory[self.user_id] = list(self.memory[self.user_id][1:]) + [action]\n",
    "            else:\n",
    "                self.memory[self.user_id] = list(self.memory[self.user_id][1:]) + [action[0]]\n",
    "                \n",
    "        if len(self.viewed_items) == len(self.related_items):\n",
    "            done = 1\n",
    "        else:\n",
    "            done = 0\n",
    "            \n",
    "        if buffer is not None:\n",
    "            buffer.push(np.array([initial_user]), np.array(initial_memory), to_np(action_emb)[0], \n",
    "                        np.array([reward]), np.array([self.user_id]), self.memory[[self.user_id], :], np.array([reward]))\n",
    "\n",
    "        return torch.tensor([self.user_id]), torch.tensor(self.memory[[self.user_id], :]), reward, done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Li8nNg-3ixZ_"
   },
   "source": [
    "## 2. Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall model\n",
    "\n",
    "<img src=\"img/full_model.png\" width=\"500\" height=\"350\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class Actor_DRR(nn.Module):\n",
    "    def __init__(self, embedding_dim, hidden_dim):\n",
    "        super().__init__()\n",
    "    \n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(embedding_dim * 3, hidden_dim),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_dim, embedding_dim)\n",
    "        )\n",
    "        \n",
    "        self.initialize()\n",
    "\n",
    "    def initialize(self):\n",
    "        for layer in self.layers:\n",
    "            if isinstance(layer, nn.Linear):\n",
    "                nn.init.kaiming_uniform_(layer.weight)\n",
    "\n",
    "    def forward(self, state):\n",
    "        return self.layers(state)\n",
    "    \n",
    "    def get_action(self, user, memory, state_repr, \n",
    "                   action_emb,\n",
    "                   items=torch.tensor([i for i in range(item_num)]),\n",
    "                   return_scores=False\n",
    "                  ):\n",
    "        state = state_repr(user, memory)\n",
    "        scores = torch.bmm(state_repr.item_embeddings(items).unsqueeze(0), \n",
    "                         action_emb.T.unsqueeze(0)).squeeze(0)\n",
    "        if return_scores:\n",
    "            return scores, torch.gather(items, 0, scores.argmax(0))\n",
    "        else:\n",
    "            return torch.gather(items, 0, scores.argmax(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class Critic_DRR(nn.Module):\n",
    "    def __init__(self, state_repr_dim, action_emb_dim, hidden_dim):\n",
    "        super().__init__()\n",
    "\n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(state_repr_dim + action_emb_dim, hidden_dim), \n",
    "            nn.ReLU(), \n",
    "            nn.Linear(hidden_dim, 1)\n",
    "        )\n",
    "\n",
    "        self.initialize()\n",
    "        \n",
    "    def initialize(self):\n",
    "        for layer in self.layers:\n",
    "            if isinstance(layer, nn.Linear):\n",
    "                nn.init.kaiming_uniform_(layer.weight)\n",
    "        \n",
    "    def forward(self, state, action):\n",
    "        x = torch.cat([state, action], 1)\n",
    "        x = self.layers(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### State representation\n",
    "\n",
    "<img src=\"img/state_representation.png\" width=\"350\" height=\"250\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class State_Repr_Module(nn.Module):\n",
    "    def __init__(self, user_num, item_num, embedding_dim, hidden_dim):\n",
    "        super().__init__()\n",
    "        self.user_embeddings = nn.Embedding(user_num, embedding_dim)\n",
    "        self.item_embeddings = nn.Embedding(item_num+1, embedding_dim, padding_idx=int(item_num))\n",
    "        self.drr_ave = torch.nn.Conv1d(in_channels=params['N'], out_channels=1, kernel_size=1)\n",
    "        \n",
    "        self.initialize()\n",
    "            \n",
    "    def initialize(self):\n",
    "        nn.init.normal_(self.user_embeddings.weight, std=0.01)\n",
    "        nn.init.normal_(self.item_embeddings.weight, std=0.01)\n",
    "        self.item_embeddings.weight.data[-1].zero_()\n",
    "        nn.init.uniform_(self.drr_ave.weight)\n",
    "        self.drr_ave.bias.data.zero_()\n",
    "\n",
    "    def forward(self, user, memory):\n",
    "        user_embedding = self.user_embeddings(user.long())\n",
    "\n",
    "        item_embeddings = self.item_embeddings(memory.long())\n",
    "        drr_ave = self.drr_ave(item_embeddings).squeeze(1)\n",
    "        \n",
    "        return torch.cat((user_embedding, user_embedding * drr_ave, drr_ave), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For evaluation we take 1 positive and 99 sampled negatives items per batch, select 10 items with best scores and calculate hit_rate@10 and nDCG@10.\n",
    "During training we choose user 6039 and track `hit` and `dcg` only for him (for evaluation speed). Final scores was computed on the whole test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resetting dataset\n",
      "Resetting dataset\n"
     ]
    }
   ],
   "source": [
    "valid_dataset = EvalDataset(\n",
    "    np.array(test_data)[np.array(test_data)[:, 0] == 6039], \n",
    "    item_num, \n",
    "    test_matrix)\n",
    "valid_loader = td.DataLoader(valid_dataset, batch_size=100, shuffle=False)\n",
    "\n",
    "full_dataset = EvalDataset(np.array(test_data), item_num, test_matrix)\n",
    "full_loader = td.DataLoader(full_dataset, batch_size=100, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_evaluation(net, state_representation, training_env_memory, loader=valid_loader):\n",
    "    hits = []\n",
    "    dcgs = []\n",
    "    test_env = Env(test_matrix)\n",
    "    test_env.memory = training_env_memory.copy()\n",
    "    user, memory = test_env.reset(int(to_np(next(iter(valid_loader))['user'])[0]))\n",
    "    for batch in loader:\n",
    "        action_emb = net(state_repr(user, memory))\n",
    "        scores, action = net.get_action(\n",
    "            batch['user'], \n",
    "            torch.tensor(test_env.memory[to_np(batch['user']).astype(int), :]), \n",
    "            state_representation, \n",
    "            action_emb,\n",
    "            batch['item'].long(), \n",
    "            return_scores=True\n",
    "        )\n",
    "        user, memory, reward, done = test_env.step(action)\n",
    "\n",
    "        _, ind = scores[:, 0].topk(10)\n",
    "        predictions = torch.take(batch['item'], ind).cpu().numpy().tolist()\n",
    "        actual = batch['item'][0].item()\n",
    "        hits.append(hit_metric(predictions, actual))\n",
    "        dcgs.append(dcg_metric(predictions, actual))\n",
    "        \n",
    "    return np.mean(hits), np.mean(dcgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "code_folding": [],
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ranger optimizer loaded. \n",
      "Gradient Centralization usage = True\n",
      "GC applied to both conv and fc layers\n",
      "Ranger optimizer loaded. \n",
      "Gradient Centralization usage = True\n",
      "GC applied to both conv and fc layers\n",
      "Ranger optimizer loaded. \n",
      "Gradient Centralization usage = True\n",
      "GC applied to both conv and fc layers\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(2)\n",
    "\n",
    "state_repr = State_Repr_Module(user_num, item_num, params['embedding_dim'], params['hidden_dim'])\n",
    "policy_net = Actor_DRR(params['embedding_dim'], params['hidden_dim'])\n",
    "value_net  = Critic_DRR(params['embedding_dim'] * 3, params['embedding_dim'], params['hidden_dim'])\n",
    "replay_buffer = Prioritized_Buffer(params['buffer_size'])\n",
    "\n",
    "target_value_net  = Critic_DRR(params['embedding_dim'] * 3, params['embedding_dim'], params['hidden_dim'])\n",
    "target_policy_net = Actor_DRR(params['embedding_dim'], params['hidden_dim'])\n",
    "\n",
    "for target_param, param in zip(target_value_net.parameters(), value_net.parameters()):\n",
    "    target_param.data.copy_(param.data)\n",
    "\n",
    "for target_param, param in zip(target_policy_net.parameters(), policy_net.parameters()):\n",
    "    target_param.data.copy_(param.data)\n",
    "\n",
    "value_criterion  = nn.MSELoss()\n",
    "value_optimizer  = Ranger(value_net.parameters(),  lr=params['value_lr'], \n",
    "                          weight_decay=params['value_decay'])\n",
    "policy_optimizer = Ranger(policy_net.parameters(), lr=params['policy_lr'], \n",
    "                          weight_decay=params['policy_decay'])\n",
    "state_repr_optimizer = Ranger(state_repr.parameters(), lr=params['state_repr_lr'], \n",
    "                              weight_decay=params['state_repr_decay'])\n",
    "\n",
    "writer = SummaryWriter(log_dir=params['log_dir'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def ddpg_update(training_env, \n",
    "                step=0,\n",
    "                batch_size=params['batch_size'], \n",
    "                gamma=params['gamma'],\n",
    "                min_value=params['min_value'],\n",
    "                max_value=params['max_value'],\n",
    "                soft_tau=params['soft_tau'],\n",
    "               ):\n",
    "    beta = get_beta(step)\n",
    "    user, memory, action, reward, next_user, next_memory, done = replay_buffer.sample(batch_size, beta)\n",
    "    user        = torch.FloatTensor(user)\n",
    "    memory      = torch.FloatTensor(memory)\n",
    "    action      = torch.FloatTensor(action)\n",
    "    reward      = torch.FloatTensor(reward)\n",
    "    next_user   = torch.FloatTensor(next_user)\n",
    "    next_memory = torch.FloatTensor(next_memory)\n",
    "    done = torch.FloatTensor(done)\n",
    "    \n",
    "    state       = state_repr(user, memory)\n",
    "    policy_loss = value_net(state, policy_net(state))\n",
    "    policy_loss = -policy_loss.mean()\n",
    "    \n",
    "    next_state     = state_repr(next_user, next_memory)\n",
    "    next_action    = target_policy_net(next_state)\n",
    "    target_value   = target_value_net(next_state, next_action.detach())\n",
    "    expected_value = reward + (1.0 - done) * gamma * target_value\n",
    "    expected_value = torch.clamp(expected_value, min_value, max_value)\n",
    "\n",
    "    value = value_net(state, action)\n",
    "    value_loss = value_criterion(value, expected_value.detach())\n",
    "    \n",
    "    state_repr_optimizer.zero_grad()\n",
    "    policy_optimizer.zero_grad()\n",
    "    policy_loss.backward(retain_graph=True)\n",
    "    policy_optimizer.step()\n",
    "\n",
    "    value_optimizer.zero_grad()\n",
    "    value_loss.backward(retain_graph=True)\n",
    "    value_optimizer.step()\n",
    "    state_repr_optimizer.step()\n",
    "\n",
    "    for target_param, param in zip(target_value_net.parameters(), value_net.parameters()):\n",
    "                target_param.data.copy_(\n",
    "                    target_param.data * (1.0 - soft_tau) + param.data * soft_tau\n",
    "                )\n",
    "\n",
    "    for target_param, param in zip(target_policy_net.parameters(), policy_net.parameters()):\n",
    "            target_param.data.copy_(\n",
    "                target_param.data * (1.0 - soft_tau) + param.data * soft_tau\n",
    "            )\n",
    "\n",
    "    writer.add_histogram('value', value, step)\n",
    "    writer.add_histogram('target_value', target_value, step)\n",
    "    writer.add_histogram('expected_value', expected_value, step)\n",
    "    writer.add_histogram('policy_loss', policy_loss, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "code_folding": [],
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4699/4699 [5:34:55<00:00,  4.28s/it]   \n"
     ]
    }
   ],
   "source": [
    "np.random.seed(16)\n",
    "train_env = Env(train_matrix)\n",
    "hits, dcgs = [], []\n",
    "hits_all, dcgs_all = [], []\n",
    "step, best_step = 0, 0\n",
    "step, best_step, best_step_all = 0, 0, 0\n",
    "users = np.random.permutation(appropriate_users)\n",
    "ou_noise = OUNoise(params['embedding_dim'], decay_period=10)\n",
    "\n",
    "for u in tqdm.tqdm(users):\n",
    "    user, memory = train_env.reset(u)\n",
    "    if params['ou_noise']:\n",
    "        ou_noise.reset()\n",
    "    for t in range(int(train_matrix[u].sum())):\n",
    "        action_emb = policy_net(state_repr(user, memory))\n",
    "        if params['ou_noise']:\n",
    "            action_emb = ou_noise.get_action(action_emb.detach().cpu().numpy()[0], t)\n",
    "        action = policy_net.get_action(\n",
    "            user, \n",
    "            torch.tensor(train_env.memory[to_np(user).astype(int), :]), \n",
    "            state_repr, \n",
    "            action_emb,\n",
    "            torch.tensor(\n",
    "                [item for item in train_env.available_items \n",
    "                if item not in train_env.viewed_items]\n",
    "            ).long()\n",
    "        )\n",
    "        user, memory, reward, done = train_env.step(\n",
    "            action, \n",
    "            action_emb,\n",
    "            buffer=replay_buffer\n",
    "        )\n",
    "\n",
    "        if len(replay_buffer) > params['batch_size']:\n",
    "            ddpg_update(train_env, step=step)\n",
    "\n",
    "        if step % 100 == 0 and step > 0:\n",
    "            hit, dcg = run_evaluation(policy_net, state_repr, train_env.memory)\n",
    "            writer.add_scalar('hit', hit, step)\n",
    "            writer.add_scalar('dcg', dcg, step)\n",
    "            hits.append(hit)\n",
    "            dcgs.append(dcg)\n",
    "            if np.mean(np.array([hit, dcg]) - np.array([hits[best_step], dcgs[best_step]])) > 0:\n",
    "                best_step = step // 100\n",
    "                torch.save(policy_net.state_dict(), params['log_dir'] + 'policy_net.pth')\n",
    "                torch.save(value_net.state_dict(), params['log_dir'] + 'value_net.pth')\n",
    "                torch.save(state_repr.state_dict(), params['log_dir'] + 'state_repr.pth')\n",
    "        if step % 10000 == 0 and step > 0:\n",
    "            hit, dcg = run_evaluation(policy_net, state_repr, train_env.memory, full_loader)\n",
    "            writer.add_scalar('hit_all', hit, step)\n",
    "            writer.add_scalar('dcg_all', dcg, step)\n",
    "            hits_all.append(hit)\n",
    "            dcgs_all.append(dcg)\n",
    "            if np.mean(np.array([hit, dcg]) - np.array([hits_all[best_step_all], dcgs_all[best_step_all]])) > 0:\n",
    "                best_step_all = step // 10000\n",
    "                torch.save(policy_net.state_dict(), params['log_dir'] + 'best_policy_net.pth')\n",
    "                torch.save(value_net.state_dict(), params['log_dir'] + 'best_value_net.pth')\n",
    "                torch.save(state_repr.state_dict(), params['log_dir'] + 'best_state_repr.pth')\n",
    "        step += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(policy_net.state_dict(), params['log_dir'] + 'policy_net_final.pth')\n",
    "torch.save(value_net.state_dict(), params['log_dir'] + 'value_net_final.pth')\n",
    "torch.save(state_repr.state_dict(), params['log_dir'] + 'state_repr_final.pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we need memory for validation, so it's better to save it and not wait next time \n",
    "with open('logs/memory.pickle', 'wb') as f:\n",
    "    pickle.dump(train_env.memory, f)\n",
    "    \n",
    "with open('logs/memory.pickle', 'rb') as f:\n",
    "    memory = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weights and logs are stored in [this folder](https://drive.google.com/drive/folders/1hsGjh8oHN4uyCmp_wtAyVPTR76ylVVgH?usp=sharing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hit rate:  0.45371301632835115 dcg:  0.25425667195713686\n"
     ]
    }
   ],
   "source": [
    "no_ou_state_repr = State_Repr_Module(user_num, item_num, params['embedding_dim'], params['hidden_dim'])\n",
    "no_ou_policy_net = Actor_DRR(params['embedding_dim'], params['hidden_dim'])\n",
    "no_ou_state_repr.load_state_dict(torch.load('logs/no_ou/' + 'best_state_repr.pth'))\n",
    "no_ou_policy_net.load_state_dict(torch.load('logs/no_ou/' + 'best_policy_net.pth'))\n",
    "    \n",
    "hit, dcg = run_evaluation(no_ou_policy_net, no_ou_state_repr, memory, full_loader)\n",
    "print('hit rate: ', hit, 'dcg: ', dcg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hit rate:  0.5024706798086426 dcg:  0.28013841397710176\n"
     ]
    }
   ],
   "source": [
    "ou_state_repr = State_Repr_Module(user_num, item_num, params['embedding_dim'], params['hidden_dim'])\n",
    "ou_policy_net = Actor_DRR(params['embedding_dim'], params['hidden_dim'])\n",
    "ou_state_repr.load_state_dict(torch.load('logs/ou_noise_04/' + 'best_state_repr.pth'))\n",
    "ou_policy_net.load_state_dict(torch.load('logs/ou_noise_04/' + 'best_policy_net.pth'))\n",
    "\n",
    "hit, dcg = run_evaluation(ou_policy_net, ou_state_repr, memory, full_loader)\n",
    "print('hit rate: ', hit, 'dcg: ', dcg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example of trained agents behaviour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's choose random user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2169\n"
     ]
    }
   ],
   "source": [
    "random_user = np.random.randint(user_num)\n",
    "print(random_user)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>genre</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>Nixon (1995)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>53</td>\n",
       "      <td>Lamerica (1994)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>55</td>\n",
       "      <td>Georgia (1995)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>72</td>\n",
       "      <td>Kicking and Screaming (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>125</td>\n",
       "      <td>Flirting With Disaster (1996)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>128</td>\n",
       "      <td>Jupiter's Wife (1994)</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217</th>\n",
       "      <td>219</td>\n",
       "      <td>Cure, The (1995)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>386</td>\n",
       "      <td>S.F.W. (1994)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>393</td>\n",
       "      <td>Street Fighter (1994)</td>\n",
       "      <td>Action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>939</th>\n",
       "      <td>951</td>\n",
       "      <td>His Girl Friday (1940)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1015</th>\n",
       "      <td>1028</td>\n",
       "      <td>Mary Poppins (1964)</td>\n",
       "      <td>Children's|Comedy|Musical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1064</th>\n",
       "      <td>1080</td>\n",
       "      <td>Monty Python's Life of Brian (1979)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134</th>\n",
       "      <td>1150</td>\n",
       "      <td>Return of Martin Guerre, The (Retour de Martin...</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1169</th>\n",
       "      <td>1186</td>\n",
       "      <td>Sex, Lies, and Videotape (1989)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1282</th>\n",
       "      <td>1302</td>\n",
       "      <td>Field of Dreams (1989)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1324</th>\n",
       "      <td>1345</td>\n",
       "      <td>Carrie (1976)</td>\n",
       "      <td>Horror</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1383</th>\n",
       "      <td>1406</td>\n",
       "      <td>C�r�monie, La (1995)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1395</th>\n",
       "      <td>1419</td>\n",
       "      <td>Walkabout (1971)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1500</th>\n",
       "      <td>1538</td>\n",
       "      <td>Second Jungle Book: Mowgli &amp; Baloo, The (1997)</td>\n",
       "      <td>Adventure|Children's</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1544</th>\n",
       "      <td>1585</td>\n",
       "      <td>Love Serenade (1996)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1599</th>\n",
       "      <td>1645</td>\n",
       "      <td>Devil's Advocate, The (1997)</td>\n",
       "      <td>Crime|Horror|Mystery|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1978</th>\n",
       "      <td>2047</td>\n",
       "      <td>Gnome-Mobile, The (1967)</td>\n",
       "      <td>Children's</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                               name  \\\n",
       "13      14                                       Nixon (1995)   \n",
       "52      53                                    Lamerica (1994)   \n",
       "54      55                                     Georgia (1995)   \n",
       "71      72                       Kicking and Screaming (1995)   \n",
       "123    125                      Flirting With Disaster (1996)   \n",
       "126    128                              Jupiter's Wife (1994)   \n",
       "217    219                                   Cure, The (1995)   \n",
       "382    386                                      S.F.W. (1994)   \n",
       "389    393                              Street Fighter (1994)   \n",
       "939    951                             His Girl Friday (1940)   \n",
       "1015  1028                                Mary Poppins (1964)   \n",
       "1064  1080                Monty Python's Life of Brian (1979)   \n",
       "1134  1150  Return of Martin Guerre, The (Retour de Martin...   \n",
       "1169  1186                    Sex, Lies, and Videotape (1989)   \n",
       "1282  1302                             Field of Dreams (1989)   \n",
       "1324  1345                                      Carrie (1976)   \n",
       "1383  1406                               C�r�monie, La (1995)   \n",
       "1395  1419                                   Walkabout (1971)   \n",
       "1500  1538     Second Jungle Book: Mowgli & Baloo, The (1997)   \n",
       "1544  1585                               Love Serenade (1996)   \n",
       "1599  1645                       Devil's Advocate, The (1997)   \n",
       "1978  2047                           Gnome-Mobile, The (1967)   \n",
       "\n",
       "                              genre  \n",
       "13                            Drama  \n",
       "52                            Drama  \n",
       "54                            Drama  \n",
       "71                     Comedy|Drama  \n",
       "123                          Comedy  \n",
       "126                     Documentary  \n",
       "217                           Drama  \n",
       "382                           Drama  \n",
       "389                          Action  \n",
       "939                          Comedy  \n",
       "1015      Children's|Comedy|Musical  \n",
       "1064                         Comedy  \n",
       "1134                          Drama  \n",
       "1169                          Drama  \n",
       "1282                          Drama  \n",
       "1324                         Horror  \n",
       "1383                          Drama  \n",
       "1395                          Drama  \n",
       "1500           Adventure|Children's  \n",
       "1544                         Comedy  \n",
       "1599  Crime|Horror|Mystery|Thriller  \n",
       "1978                     Children's  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies = pd.read_csv('data/movies.dat', sep='::', header=None, engine='python', names=['id', 'name', 'genre'])\n",
    "# in the code numeration starts with 0\n",
    "movies[movies['id'].isin(np.argwhere(test_matrix[random_user] > 0)[:, 1] + 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example we can recommend \"Nixon\" and \"Love Serenade\" and see next 3 predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([124]), tensor([127]), tensor([1228])]\n",
      "[tensor([71]), tensor([1967]), tensor([1405])]\n"
     ]
    }
   ],
   "source": [
    "predictions = []\n",
    "\n",
    "for model, state_representation in zip([ou_policy_net, no_ou_policy_net], [ou_state_repr, no_ou_state_repr]):\n",
    "    example_env = Env(test_matrix)\n",
    "    user, memory = example_env.reset(random_user)\n",
    "\n",
    "    user, memory, reward, _ = example_env.step(torch.tensor([13]))\n",
    "    user, memory, reward, _ = example_env.step(torch.tensor([1584]))\n",
    "    preds = []\n",
    "    for _ in range(3):\n",
    "        action_emb = model(state_representation(user, memory))\n",
    "        action = model.get_action(\n",
    "            user, \n",
    "            torch.tensor(example_env.memory[to_np(user).astype(int), :]), \n",
    "            state_representation, \n",
    "            action_emb,\n",
    "            torch.tensor(\n",
    "                [item for item in example_env.available_items \n",
    "                if item not in example_env.viewed_items]\n",
    "            ).long()\n",
    "        )\n",
    "        user, memory, reward, _ = example_env.step(action)\n",
    "        preds.append(action)\n",
    "\n",
    "    predictions.append(preds)\n",
    "\n",
    "print(predictions[0])\n",
    "print(predictions[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model trained with OU noise recommended related `Comedy` and `Documentary` after that switch recommendations to nonrelated `Crime|Film-Noir|Thriller`.\n",
    "Model trained without OU noise recommended `Comedy|Drama`, `Children's|Comedy`, `Drama` (two of them are related).\n",
    "\n",
    "Both models seems to be reasonable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training process logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=img/learning_curve.png>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logs are consistent with expectations. Adding noise increase metrics (std=0.4 performs the best, after 0.6 model starts to degrade)."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "s-iWfb5TixZc",
    "baIIDXdAixaH"
   ],
   "name": "pytorch.pipelines.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}