LSTM_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hello everyone,\n",
    "\n",
    "##### Welcome in this place.\n",
    "\n",
    "This notebook is a really simple baseline of Deep Learning on Sentiment Analysis. In this notebook I'll show you a simple preprocessing and a simple neural net using a Long Short-Term Memory (LSTM) layer.\n",
    "\n",
    "The goal of this notebook is to predict the sentiment of feedbacks of movies. The label we have is \"0\" for bad movies (mark < 5 IMDB) and \"1\" for good movies (mark > 7 IMDB)\n",
    "\n",
    "#### STRATEGY:\n",
    "\n",
    "In this notebook i'm going to:\n",
    "\n",
    "    1) load data\n",
    "    2) clean data\n",
    "    3) train word2vec ad hoc\n",
    "    4) creat extra-variable\n",
    "    5) train model\n",
    "\n",
    "In a previous notebook I've showed a really simple introduction to word2vec. \n",
    "If you want to check what I've previously done or if you just need a refreshment about word2vec you can find that example here: https://lnkd.in/dddZpKt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "import time \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.models import word2vec, KeyedVectors, Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"With all this stuff going down at the moment with MJ i\\'ve started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ\\'s feeling towards the press and also the obvious message of drugs are bad m\\'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci\\'s character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ\\'s music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ\\'s bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i\\'ve gave this subject....hmmm well i don\\'t know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(\"prova.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "data.review[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eng_stop = nltk.corpus.stopwords.words('english')\n",
    "\n",
    "def nostop(text, stops):\n",
    "    pattern = re.compile(r'\\b(' + r'|'.join(stops) + r')\\b\\s*')\n",
    "    text = pattern.sub(' ', text)\n",
    "    text = re.sub(\" +\", \" \", text).strip()\n",
    "    return(text)\n",
    "\n",
    "def fast_clean(x):\n",
    "    out = re.sub('[^A-Za-z0-9]+', ' ', x)\n",
    "    out = out.lower()\n",
    "    return(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stuff going moment mj started listening music watching odd documentary watched wiz watched moonwalker maybe want get certain insight guy thought really cool eighties maybe make mind whether guilty innocent moonwalker part biography part feature film remember going see cinema originally released subtle messages mj feeling towards press also obvious message drugs bad kay br br visually impressive course michael jackson unless remotely like mj anyway going hate find boring may call mj egotist consenting making movie mj fans would say made fans true really nice br br actual feature film bit finally starts 20 minutes excluding smooth criminal sequence joe pesci convincing psychopathic powerful drug lord wants mj dead bad beyond mj overheard plans nah joe pesci character ranted wanted people know supplying drugs etc dunno maybe hates mj music br br lots cool things like mj turning car robot whole speed demon sequence also director must patience saint came filming kiddy bad sequence usually directors hate working one kid let alone whole bunch performing complex dance scene br br bottom line movie people like mj one level another think people stay away try give wholesome message ironically mj bestest buddy movie girl michael jackson truly one talented people ever grace planet guilty well attention gave subject hmmm well know people different behind closed doors know fact either extremely nice stupid guy one sickest liars hope latter'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['review_clean_nostop'] = data.apply(axis=1, func = lambda x: nostop(fast_clean(x.review),eng_stop))\n",
    "data['review_clean_nostop'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_nltk = [i for i in data['review_clean_nostop']]\n",
    "\n",
    "corpus_nltk = \" \".join(corpus_nltk)\n",
    "token_nltk = nltk.word_tokenize(corpus_nltk)\n",
    "\n",
    "freqd_nltk = nltk.FreqDist(token_nltk)\n",
    "freqd_nltk_L = dict((k, v) for k, v in freqd_nltk.items() if v == 1)\n",
    "dropL = list(freqd_nltk_L.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I worked for 12.44 minutes\n"
     ]
    }
   ],
   "source": [
    "x = time.time()\n",
    "data['review_clean_nostop_droplow'] = data.apply(axis=1, func = lambda x: nostop(x.review_clean_nostop,dropL))\n",
    "print('I worked for ' + str(round((time.time()-x)/60,2)) + ' minutes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFbRJREFUeJzt3W+MnWeZ3/Hvb00I5k9K0sy6xja1WZlKTqQ1xHLdsqoo\nKRs3IBzeIKNCvGoaIyVF0CJRG6QuvLAUWv5so5ZUAbJxdlkia4HGgmS7IcsKITUxEzbEsYMbUyeN\np048gKihlbzrcPXFuW2eDnbmjGc858TP9yMdnftcz3OfuY4T+zfPv/OkqpAk9dNvjLoBSdLoGAKS\n1GOGgCT1mCEgST1mCEhSjxkCktRjhoAk9ZghIEk9ZghIUo+9bNQNzObKK6+s1atXj7oNSXpJefTR\nR39cVROzrTf2IbB69WomJydH3YYkvaQkeWaY9dwdJEk9ZghIUo8ZApLUY4aAJPWYISBJPWYISFKP\nGQKS1GOGgCT1mCEgST029lcML6bVO755Zvz0be8YYSeStDjcEpCkHjMEJKnHhg6BJEuS/FWSb7TX\nVyR5MMlT7fnyzro7kxxOcijJdZ36NUn2t2W3J8nCfhxJ0lzMZUvgQ8CTndc7gIeqai3wUHtNknXA\nVuAqYDPw+SRL2pw7gJuBte2xeV7dS5LmZagQSLISeAfwxU55C7C7jXcDN3Tq91bVyao6AhwGNiZZ\nDlxWVQ9XVQH3dOZIkkZg2C2BPwA+CvyyU1tWVcfa+DlgWRuvAJ7trHe01Va08cz6r0myPclkksnp\n6ekhW5QkzdWsIZDkncDxqnr0XOu03+xroZqqqjurakNVbZiYmPXGOJKk8zTMdQJvAd6V5HrgFcBl\nSf4YeD7J8qo61nb1HG/rTwGrOvNXttpUG8+sS5JGZNYtgaraWVUrq2o1gwO+f1FV7wP2AtvaatuA\n+9p4L7A1yaVJ1jA4ALyv7To6kWRTOyvoxs4cSdIIzOeK4duAPUluAp4B3gNQVQeS7AEOAqeAW6vq\nhTbnFuBuYCnwQHtIkkZkTiFQVX8J/GUb/wS49hzr7QJ2naU+CVw91yYlSReGVwxLUo8ZApLUY4aA\nJPWYISBJPWYISFKPGQKS1GOGgCT1mCEgST1mCEhSjxkCktRjhoAk9ZghIEk9ZghIUo8ZApLUY4aA\nJPWYISBJPTbMjeZfkWRfkh8kOZDkk63+iSRTSR5rj+s7c3YmOZzkUJLrOvVrkuxvy25vt5mUJI3I\nMHcWOwm8rap+keQS4LtJTt8W8nNV9enuyknWMbgX8VXA64BvJXlju8XkHcDNwCPA/cBmvMWkJI3M\nMDear6r6RXt5SXvUi0zZAtxbVSer6ghwGNiYZDlwWVU9XFUF3APcML/2JUnzMdQxgSRLkjwGHAce\nrKpH2qIPJnk8yV1JLm+1FcCznelHW21FG8+sS5JGZKgQqKoXqmo9sJLBb/VXM9i18wZgPXAM+MxC\nNZVke5LJJJPT09ML9baSpBnmdHZQVf0M+Dawuaqeb+HwS+ALwMa22hSwqjNtZatNtfHM+tl+zp1V\ntaGqNkxMTMylRUnSHAxzdtBEkte28VLg7cAP2z7+094NPNHGe4GtSS5NsgZYC+yrqmPAiSSb2llB\nNwL3LeBnkSTN0TBnBy0HdidZwiA09lTVN5L8UZL1DA4SPw18AKCqDiTZAxwETgG3tjODAG4B7gaW\nMjgryDODJGmEZg2BqnoceNNZ6u9/kTm7gF1nqU8CV8+xR0nSBeIVw5LUY4aAJPWYISBJPWYISFKP\nGQKS1GOGgCT1mCEgST1mCEhSjxkCktRjhoAk9ZghIEk9ZghIUo8ZApLUY4aAJPWYISBJPWYISFKP\nDXN7yVck2ZfkB0kOJPlkq1+R5MEkT7XnyztzdiY5nORQkus69WuS7G/Lbm+3mRxLq3d888xDki5W\nw2wJnATeVlW/DawHNifZBOwAHqqqtcBD7TVJ1gFbgauAzcDn260pAe4AbmZw3+G1bbkkaURmDYEa\n+EV7eUl7FLAF2N3qu4Eb2ngLcG9VnayqI8BhYGO7Mf1lVfVwVRVwT2eOJGkEhjomkGRJkseA48CD\nVfUIsKyqjrVVngOWtfEK4NnO9KOttqKNZ9YlSSMyVAhU1QtVtR5YyeC3+qtnLC8GWwcLIsn2JJNJ\nJqenpxfqbSVJM8zp7KCq+hnwbQb78p9vu3hoz8fbalPAqs60la021cYz62f7OXdW1Yaq2jAxMTGX\nFiVJczDM2UETSV7bxkuBtwM/BPYC29pq24D72ngvsDXJpUnWMDgAvK/tOjqRZFM7K+jGzhxJ0gi8\nbIh1lgO72xk+vwHsqapvJPlvwJ4kNwHPAO8BqKoDSfYAB4FTwK1V9UJ7r1uAu4GlwAPtIUkakVlD\noKoeB950lvpPgGvPMWcXsOss9Ung6l+fIUkaBa8YlqQeMwQkqccMAUnqMUNAknrMEJCkHjMEJKnH\nDAFJ6jFDQJJ6zBCQpB4b5msjLmreOUxSn7klIEk9ZghIUo8ZApLUY4aAJPWYISBJPWYISFKPDXN7\nyVVJvp3kYJIDST7U6p9IMpXksfa4vjNnZ5LDSQ4lua5TvybJ/rbs9nabSUnSiAxzncAp4CNV9f0k\nrwEeTfJgW/a5qvp0d+Uk64CtwFXA64BvJXlju8XkHcDNwCPA/QxuWO8tJiVpRGbdEqiqY1X1/Tb+\nOfAksOJFpmwB7q2qk1V1BDgMbEyyHLisqh6uqgLuAW6Y9yeQJJ23OR0TSLKawf2GH2mlDyZ5PMld\nSS5vtRXAs51pR1ttRRvPrEuSRmToEEjyauCrwIer6gSDXTtvANYDx4DPLFRTSbYnmUwyOT09vVBv\nK0maYagQSHIJgwD4clV9DaCqnq+qF6rql8AXgI1t9SlgVWf6ylabauOZ9V9TVXdW1Yaq2jAxMTGX\nzyNJmoNhzg4K8CXgyar6bKe+vLPau4En2ngvsDXJpUnWAGuBfVV1DDiRZFN7zxuB+xboc0iSzsMw\nZwe9BXg/sD/JY632MeC9SdYDBTwNfACgqg4k2QMcZHBm0a3tzCCAW4C7gaUMzgryzCBJGqFZQ6Cq\nvguc7Xz++19kzi5g11nqk8DVc2lQknTheMWwJPWYISBJPWYISFKPGQKS1GOGgCT1mCEgST1mCEhS\njxkCktRjhoAk9ZghIEk9ZghIUo8ZApLUY4aAJPXYMF8l3Xurd3zzzPjp294xwk4kaWG5JSBJPWYI\nSFKPDXN7yVVJvp3kYJIDST7U6lckeTDJU+358s6cnUkOJzmU5LpO/Zok+9uy29ttJiVJIzLMlsAp\n4CNVtQ7YBNyaZB2wA3ioqtYCD7XXtGVbgauAzcDnkyxp73UHcDOD+w6vbcslSSMyawhU1bGq+n4b\n/xx4ElgBbAF2t9V2Aze08Rbg3qo6WVVHgMPAxnZj+suq6uGqKuCezhxJ0gjM6ZhAktXAm4BHgGVV\ndawteg5Y1sYrgGc704622oo2nlmXJI3I0CGQ5NXAV4EPV9WJ7rL2m30tVFNJtieZTDI5PT29UG8r\nSZphqBBIcgmDAPhyVX2tlZ9vu3hoz8dbfQpY1Zm+stWm2nhm/ddU1Z1VtaGqNkxMTAz7WSRJczTM\n2UEBvgQ8WVWf7SzaC2xr423AfZ361iSXJlnD4ADwvrbr6ESSTe09b+zMkSSNwDBXDL8FeD+wP8lj\nrfYx4DZgT5KbgGeA9wBU1YEke4CDDM4surWqXmjzbgHuBpYCD7SHJGlEZg2BqvoucK7z+a89x5xd\nwK6z1CeBq+fSoCTpwvGKYUnqMUNAknrMEJCkHjMEJKnHDAFJ6jFDQJJ6zBCQpB4zBCSpx7zH8Bx5\nv2FJFxO3BCSpxwwBSeoxQ0CSeswQkKQeMwQkqccMAUnqMUNAknpsmNtL3pXkeJInOrVPJJlK8lh7\nXN9ZtjPJ4SSHklzXqV+TZH9bdnu7xaQkaYSG2RK4G9h8lvrnqmp9e9wPkGQdsBW4qs35fJIlbf07\ngJsZ3HN47TneU5K0iGYNgar6DvDTId9vC3BvVZ2sqiPAYWBjkuXAZVX1cFUVcA9ww/k2LUlaGPM5\nJvDBJI+33UWXt9oK4NnOOkdbbUUbz6xLkkbofEPgDuANwHrgGPCZBesISLI9yWSSyenp6YV8a0lS\nx3mFQFU9X1UvVNUvgS8AG9uiKWBVZ9WVrTbVxjPr53r/O6tqQ1VtmJiYOJ8WJUlDOK8QaPv4T3s3\ncPrMob3A1iSXJlnD4ADwvqo6BpxIsqmdFXQjcN88+pYkLYBZv0o6yVeAtwJXJjkK/D7w1iTrgQKe\nBj4AUFUHkuwBDgKngFur6oX2VrcwONNoKfBAe0iSRmjWEKiq956l/KUXWX8XsOss9Ung6jl1J0m6\noLxiWJJ6zBCQpB4zBCSpxwwBSeoxQ0CSeswQkKQeMwQkqccMAUnqMUNAknps1iuGdW6rd3zzzPjp\n294xwk4k6fz0MgS6/3hLUp+5O0iSeswQkKQe683uoAu9C8jjA5JeitwSkKQeMwQkqcdmDYEkdyU5\nnuSJTu2KJA8meao9X95ZtjPJ4SSHklzXqV+TZH9bdnu7zaQkaYSG2RK4G9g8o7YDeKiq1gIPtdck\nWQdsBa5qcz6fZEmbcwdwM4P7Dq89y3tKkhbZrCFQVd8BfjqjvAXY3ca7gRs69Xur6mRVHQEOAxvb\njekvq6qHq6qAezpzJEkjcr7HBJZV1bE2fg5Y1sYrgGc76x1ttRVtPLMuSRqheR8Ybr/Z1wL0ckaS\n7Ukmk0xOT08v5FtLkjrONwSeb7t4aM/HW30KWNVZb2WrTbXxzPpZVdWdVbWhqjZMTEycZ4uSpNmc\nbwjsBba18Tbgvk59a5JLk6xhcAB4X9t1dCLJpnZW0I2dOZKkEZn1iuEkXwHeClyZ5Cjw+8BtwJ4k\nNwHPAO8BqKoDSfYAB4FTwK1V9UJ7q1sYnGm0FHigPSRJIzRrCFTVe8+x6NpzrL8L2HWW+iRw9Zy6\nkyRdUF4xLEk9ZghIUo8ZApLUY735KunF5NdKS3qpcEtAknrMLYELzK0CSePMLQFJ6jFDQJJ6zBCQ\npB4zBCSpxwwBSeoxQ0CSeswQkKQe8zqBReQ1A5LGjVsCktRjhoAk9di8QiDJ00n2J3ksyWSrXZHk\nwSRPtefLO+vvTHI4yaEk1823eUnS/CzEMYF/XFU/7rzeATxUVbcl2dFe/5sk64CtwFXA64BvJXlj\n5/aTvdI9PgAeI5A0Ghdid9AWYHcb7wZu6NTvraqTVXUEOAxsvAA/X5I0pPmGQDH4jf7RJNtbbVlV\nHWvj54BlbbwCeLYz92irSZJGZL67g36nqqaS/CbwYJIfdhdWVSWpub5pC5TtAK9//evn2aIk6Vzm\ntSVQVVPt+TjwdQa7d55PshygPR9vq08BqzrTV7ba2d73zqraUFUbJiYm5tOiJOlFnHcIJHlVktec\nHgO/CzwB7AW2tdW2Afe18V5ga5JLk6wB1gL7zvfnX2xW7/jmmYckLZb57A5aBnw9yen3+ZOq+rMk\n3wP2JLkJeAZ4D0BVHUiyBzgInAJu7euZQZI0Ls47BKrqfwC/fZb6T4BrzzFnF7DrfH9mX/j1EpIW\ni1cMS1KPGQKS1GOGgCT1mF8l/RLisQJJC80tAUnqMbcExpzXDUi6kNwSkKQec0vgJepcWwgeK5A0\nF24JSFKPGQKS1GOGgCT1mMcELjIeK5A0F24JSFKPuSXQE8Ncb+DWgtQ/hoDO8GsppP65qEPAq20X\nhuEgXbwWPQSSbAb+A7AE+GJV3bbYPWh25wpQDzxLF5dFDYEkS4D/BLwdOAp8L8neqjq4mH1o4c11\nq8vQkMbDYm8JbAQOt1tTkuReYAuD+w6rR9yikMbDYofACuDZzuujwN9f5B40xuZ7HMcQkeZmLA8M\nJ9kObG8vf5Hk0Hm8zZXAjxeuqwvGPhdQPvXS6JOXyJ8n9rnQFrPPvzvMSosdAlPAqs7rla32/6mq\nO4E75/ODkkxW1Yb5vMdisM+FZZ8Lyz4X1jj2udhXDH8PWJtkTZKXA1uBvYvcgySpWdQtgao6leRf\nAv+VwSmid1XVgcXsQZL0K4t+TKCq7gfuX4QfNa/dSYvIPheWfS4s+1xYY9dnqmrUPUiSRsRvEZWk\nHrsoQyDJ5iSHkhxOsmOEfaxK8u0kB5McSPKhVr8iyYNJnmrPl3fm7Gx9H0py3SL3uyTJXyX5xrj2\nmeS1Sf40yQ+TPJnkH4xpn/+q/Td/IslXkrxiHPpMcleS40me6NTm3FeSa5Lsb8tuT5JF6PPft//u\njyf5epLXjmOfnWUfSVJJrhx1ny+qqi6qB4MDzj8C3gC8HPgBsG5EvSwH3tzGrwH+O7AO+HfAjlbf\nAXyqjde1fi8F1rTPsWQR+/3XwJ8A32ivx65PYDfwL9r45cBrx61PBhdFHgGWttd7gN8bhz6BfwS8\nGXiiU5tzX8A+YBMQ4AHgny5Cn78LvKyNPzWufbb6KgYnwDwDXDnqPl/scTFuCZz5aoqq+mvg9FdT\nLLqqOlZV32/jnwNPMvgHYguDf8xozze08Rbg3qo6WVVHgMMMPs8Fl2Ql8A7gi53yWPWZ5G8x+Ev3\nJYCq+uuq+tm49dm8DFia5GXAK4H/NQ59VtV3gJ/OKM+pryTLgcuq6uEa/At2T2fOBeuzqv68qk61\nlw8zuM5o7PpsPgd8FOgedB1Zny/mYgyBs301xYoR9XJGktXAm4BHgGVVdawteg5Y1saj7P0PGPxP\n+8tObdz6XANMA3/Ydlt9Mcmrxq3PqpoCPg38T+AY8L+r6s/Hrc+Oufa1oo1n1hfTP2fwGzOMWZ9J\ntgBTVfWDGYvGqs/TLsYQGDtJXg18FfhwVZ3oLmvJP9JTtJK8EzheVY+ea51x6JPBb9dvBu6oqjcB\n/4fB7oszxqHPtk99C4PQeh3wqiTv664zDn2ezbj21ZXk48Ap4Muj7mWmJK8EPgb821H3MqyLMQSG\n+mqKxZLkEgYB8OWq+lorP982AWnPx1t9VL2/BXhXkqcZ7D57W5I/HsM+jwJHq+qR9vpPGYTCuPX5\nT4AjVTVdVX8DfA34h2PY52lz7WuKX+2K6dYvuCS/B7wT+GctsGC8+vwtBuH/g/b3aSXw/SR/Z8z6\nPONiDIGx+WqKdoT/S8CTVfXZzqK9wLY23gbc16lvTXJpkjXAWgYHjC6oqtpZVSurajWDP6+/qKr3\njWGfzwHPJvl7rXQtg68hH6s+GewG2pTkle3/gWsZHA8atz5Pm1NfbdfRiSSb2ue7sTPngsnghlQf\nBd5VVf93Rv9j0WdV7a+q36yq1e3v01EGJ4c8N059zmz6onsA1zM4E+dHwMdH2MfvMNi0fhx4rD2u\nB/428BDwFPAt4IrOnI+3vg+xiGcIdH7+W/nV2UFj1yewHphsf6b/Bbh8TPv8JPBD4AngjxicETLy\nPoGvMDhO8TcM/oG66Xz6Aja0z/Yj4D/SLjy9wH0eZrBP/fTfpf88jn3OWP407eygUfb5Yg+vGJak\nHrsYdwdJkoZkCEhSjxkCktRjhoAk9ZghIEk9ZghIUo8ZApLUY4aAJPXY/wPPauyM+afsGgAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f68e90859b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "count    25000.000000\n",
       "mean       125.097840\n",
       "std         94.649944\n",
       "min          4.000000\n",
       "25%         66.000000\n",
       "50%         93.000000\n",
       "75%        153.000000\n",
       "max       1455.000000\n",
       "Name: len, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['len'] = data.apply(axis=1, func = lambda x: len(x.review_clean_nostop.split()))\n",
    "plt.hist(data['len'], bins=100)\n",
    "plt.show()\n",
    "data['len'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = data.sample(frac=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test = pd.read_csv(\"testData.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "unlabeled = pd.read_csv(\"unlabeledTrainData.tsv\", header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test['review_clean_nostop'] = test.apply(axis=1, func = lambda x: nostop(fast_clean(x.review),eng_stop))\n",
    "unlabeled['review_clean_nostop'] = unlabeled.apply(axis=1, func = lambda x: nostop(fast_clean(x.review),eng_stop))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I worked for 36.71 minutes\n"
     ]
    }
   ],
   "source": [
    "x = time.time()\n",
    "test['review_clean_nostop_droplow'] = test.apply(axis=1, func = lambda x: nostop(x.review_clean_nostop,dropL))\n",
    "unlabeled['review_clean_nostop_droplow'] = unlabeled.apply(axis=1, func = lambda x: nostop(x.review_clean_nostop,dropL))\n",
    "print('I worked for ' + str(round((time.time()-x)/60,2)) + ' minutes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_corpus(data, x):\n",
    "    corpus = []\n",
    "    for sentence in data[x].iteritems():\n",
    "        word_list = sentence[1].split()\n",
    "        corpus.append(word_list)\n",
    "\n",
    "    return corpus\n",
    "\n",
    "corpus = build_corpus(data, 'review_clean_nostop_droplow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_test = build_corpus(test, 'review_clean_nostop_droplow')\n",
    "corpus_unlabeled = build_corpus(unlabeled, 'review_clean_nostop_droplow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_all = corpus + corpus_test + corpus_unlabeled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I worked for 0.59 minutes\n"
     ]
    }
   ],
   "source": [
    "x = time.time()\n",
    "n_dim = 50\n",
    "model2vec = word2vec.Word2Vec(corpus_all, size=n_dim, window=5, min_count=5, workers=4)\n",
    "print('I worked for ' + str(round((time.time()-x)/60,2)) + ' minutes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def count_word(x,posne):\n",
    "    app = x.split()\n",
    "    count = 0\n",
    "    for i in app:\n",
    "        if i in posne:\n",
    "            count += 1\n",
    "    return(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pos = open('positive.txt', 'r', encoding='latin1').readlines()\n",
    "pos = \"\".join(pos).split(\"\\n\")\n",
    "\n",
    "neg = open('negative.txt', 'r', encoding='latin1').readlines()\n",
    "neg = \"\".join(neg).split(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I worked for 4.97 minutes\n"
     ]
    }
   ],
   "source": [
    "x = time.time()\n",
    "\n",
    "data['positive'] = data.apply(axis=1, func = lambda x: count_word(x.review_clean_nostop_droplow, pos))\n",
    "data['negative'] = data.apply(axis=1, func = lambda x: count_word(x.review_clean_nostop_droplow, neg))\n",
    "\n",
    "data['positive_ratio'] = data['positive']/data['len']\n",
    "data['negative_ratio'] = data['negative']/data['len']\n",
    "\n",
    "data['special_char'] = data.apply(axis=1, func = lambda x: len(re.findall(\"\\.|\\:|\\;|\\,|\\(|\\)|\\[|\\]|\\&|\\'|\\\"|\\-|\\_|\\°|\\{|\\}|\\+\\*\\#\\@\\/\",x.review)))\n",
    "data['esclamation_mark'] = data.apply(axis=1, func = lambda x: len(re.findall(\"\\!\",x.review)))\n",
    "data['question_mark'] = data.apply(axis=1, func = lambda x: len(re.findall(\"\\?\",x.review)))\n",
    "\n",
    "print('I worked for ' + str(round((time.time()-x)/60,2)) + ' minutes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# model2vec.save('w2v_movies_sentiment')\n",
    "# data.to_csv('data_sentiment_movies.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's play with KERAS!\n",
    "\n",
    "Keras is a cool neural network API written in Python. You can find all the documentation you need here: https://keras.io/#you-have-just-found-keras\n",
    "\n",
    "If you are confortable with math and you want to go deep you can find the introduction to LSTM here:\n",
    "http://www.bioinf.jku.at/publications/older/2604.pdf\n",
    "\n",
    "\n",
    "http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "\n",
    "\n",
    "Or a funny explanation here: \n",
    "http://blog.echen.me/2017/05/30/exploring-lstms/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Dense, Dropout, Activation, Input, BatchNormalization\n",
    "from keras.layers import Embedding, LSTM, concatenate, GRU\n",
    "from keras.callbacks import EarlyStopping, ReduceLROnPlateau\n",
    "from keras.optimizers import SGD\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_len = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88582 unique tokens\n",
      "Shape of data tensor: (25000, 200)\n",
      "Shape of label tensor: (25000,)\n",
      "I worked for 0.12 minutes\n"
     ]
    }
   ],
   "source": [
    "x = time.time()\n",
    "\n",
    "texts = list(data.review.values)\n",
    "\n",
    "tokenizer = Tokenizer(num_words=25000)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens' % len(word_index))\n",
    "\n",
    "data_1 = pad_sequences(sequences, maxlen=input_len)\n",
    "pred = np.array(data.sentiment.values)\n",
    "print('Shape of data tensor:', data_1.shape)\n",
    "print('Shape of label tensor:', pred.shape)\n",
    "\n",
    "print('I worked for ' + str(round((time.time()-x)/60,2)) + ' minutes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_dim = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_words = min(200000, len(word_index))+1\n",
    "\n",
    "embedding_matrix = np.zeros((nb_words, n_dim))\n",
    "for word, i in word_index.items():\n",
    "    if word in model2vec.wv.vocab:\n",
    "        embedding_matrix[i] = model2vec.wv[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_21 (InputLayer)            (None, 200)           0                                            \n",
      "____________________________________________________________________________________________________\n",
      "embedding_11 (Embedding)         (None, 200, 50)       4429150     input_21[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_58 (BatchNor (None, 200, 50)       200         embedding_11[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "dropout_49 (Dropout)             (None, 200, 50)       0           batch_normalization_58[0][0]     \n",
      "____________________________________________________________________________________________________\n",
      "lstm_11 (LSTM)                   (None, 512)           1153024     dropout_49[0][0]                 \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_59 (BatchNor (None, 512)           2048        lstm_11[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "dropout_50 (Dropout)             (None, 512)           0           batch_normalization_59[0][0]     \n",
      "____________________________________________________________________________________________________\n",
      "input_22 (InputLayer)            (None, 8)             0                                            \n",
      "____________________________________________________________________________________________________\n",
      "dense_47 (Dense)                 (None, 512)           262656      dropout_50[0][0]                 \n",
      "____________________________________________________________________________________________________\n",
      "dropout_51 (Dropout)             (None, 8)             0           input_22[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "activation_37 (Activation)       (None, 512)           0           dense_47[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "dense_48 (Dense)                 (None, 32)            288         dropout_51[0][0]                 \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_60 (BatchNor (None, 512)           2048        activation_37[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_61 (BatchNor (None, 32)            128         dense_48[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "concatenate_11 (Concatenate)     (None, 544)           0           batch_normalization_60[0][0]     \n",
      "                                                                   batch_normalization_61[0][0]     \n",
      "____________________________________________________________________________________________________\n",
      "dropout_52 (Dropout)             (None, 544)           0           concatenate_11[0][0]             \n",
      "____________________________________________________________________________________________________\n",
      "dense_49 (Dense)                 (None, 256)           139520      dropout_52[0][0]                 \n",
      "____________________________________________________________________________________________________\n",
      "activation_38 (Activation)       (None, 256)           0           dense_49[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_62 (BatchNor (None, 256)           1024        activation_38[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dropout_53 (Dropout)             (None, 256)           0           batch_normalization_62[0][0]     \n",
      "____________________________________________________________________________________________________\n",
      "dense_50 (Dense)                 (None, 128)           32896       dropout_53[0][0]                 \n",
      "____________________________________________________________________________________________________\n",
      "activation_39 (Activation)       (None, 128)           0           dense_50[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_63 (BatchNor (None, 128)           512         activation_39[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_51 (Dense)                 (None, 1)             129         batch_normalization_63[0][0]     \n",
      "____________________________________________________________________________________________________\n",
      "activation_40 (Activation)       (None, 1)             0           dense_51[0][0]                   \n",
      "====================================================================================================\n",
      "Total params: 6,023,623\n",
      "Trainable params: 1,591,493\n",
      "Non-trainable params: 4,432,130\n",
      "____________________________________________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model_input = Input(shape=(input_len,))\n",
    "\n",
    "model = Embedding(nb_words, n_dim, weights=[embedding_matrix],input_length=input_len,\n",
    "        trainable=False)(model_input)\n",
    "model = BatchNormalization()(model)\n",
    "\n",
    "model = Dropout(0.25)(model)\n",
    "\n",
    "model = LSTM(512, dropout=0.35,recurrent_dropout=0.35)(model)\n",
    "model = BatchNormalization()(model)\n",
    "\n",
    "model = Dropout(0.25)(model)\n",
    "model = Dense(512)(model)\n",
    "model = Activation('relu')(model)\n",
    "model = BatchNormalization()(model)\n",
    "\n",
    "model_input2 = Input(shape=(8,))\n",
    "\n",
    "model2 = Dropout(0.25)(model_input2)\n",
    "model2 = Dense(32, activation='relu')(model2)\n",
    "model2 = BatchNormalization()(model2)\n",
    "\n",
    "model = concatenate([model, model2])\n",
    "\n",
    "model = Dropout(0.25)(model)\n",
    "model = Dense(256)(model)\n",
    "model = Activation('relu')(model)\n",
    "model = BatchNormalization()(model)\n",
    "\n",
    "model = Dropout(0.25)(model)\n",
    "model = Dense(128)(model)\n",
    "model = Activation('relu')(model)\n",
    "model = BatchNormalization()(model)\n",
    "\n",
    "model = Dense(1)(model)\n",
    "out = Activation('sigmoid')(model)\n",
    "\n",
    "opt = SGD(lr = 0.1, momentum = 0.9, nesterov=True)\n",
    "\n",
    "model = Model(inputs=[model_input,model_input2], outputs=out)\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=opt,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "extra_variable = ['positive','negative','positive_ratio','len','negative_ratio',\n",
    "                    'special_char','esclamation_mark','question_mark']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/200\n",
      "15000/15000 [==============================] - 47s - loss: 0.5926 - acc: 0.6885 - val_loss: 0.5008 - val_acc: 0.7779\n",
      "Epoch 2/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5663 - acc: 0.7055 - val_loss: 0.7619 - val_acc: 0.5214\n",
      "Epoch 3/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5518 - acc: 0.7153 - val_loss: 0.4952 - val_acc: 0.7509\n",
      "Epoch 4/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5458 - acc: 0.7229 - val_loss: 0.5564 - val_acc: 0.7537\n",
      "Epoch 5/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5136 - acc: 0.7469 - val_loss: 0.4294 - val_acc: 0.7919\n",
      "Epoch 6/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5616 - acc: 0.7110 - val_loss: 0.5800 - val_acc: 0.6741\n",
      "Epoch 7/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.5310 - acc: 0.7357 - val_loss: 0.5995 - val_acc: 0.7537\n",
      "Epoch 8/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.4796 - acc: 0.7715 - val_loss: 0.4672 - val_acc: 0.8065\n",
      "Epoch 9/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.4219 - acc: 0.8097 - val_loss: 0.4508 - val_acc: 0.8154\n",
      "Epoch 10/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.4033 - acc: 0.8189 - val_loss: 0.4908 - val_acc: 0.8137\n",
      "Epoch 11/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3989 - acc: 0.8223 - val_loss: 0.3492 - val_acc: 0.8582\n",
      "Epoch 12/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3848 - acc: 0.8283 - val_loss: 0.3198 - val_acc: 0.8627\n",
      "Epoch 13/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3751 - acc: 0.8349 - val_loss: 0.3273 - val_acc: 0.8588\n",
      "Epoch 14/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3693 - acc: 0.8359 - val_loss: 0.3167 - val_acc: 0.8666\n",
      "Epoch 15/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3641 - acc: 0.8372 - val_loss: 0.3079 - val_acc: 0.8691\n",
      "Epoch 16/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3763 - acc: 0.8304 - val_loss: 0.3553 - val_acc: 0.8483\n",
      "Epoch 17/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3635 - acc: 0.8408 - val_loss: 0.3354 - val_acc: 0.8642\n",
      "Epoch 18/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3508 - acc: 0.8477 - val_loss: 0.3236 - val_acc: 0.8567\n",
      "Epoch 19/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3521 - acc: 0.8457 - val_loss: 0.4087 - val_acc: 0.8448\n",
      "Epoch 20/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3410 - acc: 0.8513 - val_loss: 0.3253 - val_acc: 0.8652\n",
      "Epoch 21/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3344 - acc: 0.8561 - val_loss: 0.3016 - val_acc: 0.8721\n",
      "Epoch 22/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3377 - acc: 0.8542 - val_loss: 0.3094 - val_acc: 0.8690\n",
      "Epoch 23/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3322 - acc: 0.8554 - val_loss: 0.3089 - val_acc: 0.8716\n",
      "Epoch 24/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3279 - acc: 0.8586 - val_loss: 0.3021 - val_acc: 0.8729\n",
      "Epoch 25/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3348 - acc: 0.8536 - val_loss: 0.3065 - val_acc: 0.8748\n",
      "Epoch 26/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3289 - acc: 0.8541 - val_loss: 0.3062 - val_acc: 0.8768\n",
      "Epoch 27/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3229 - acc: 0.8608 - val_loss: 0.2907 - val_acc: 0.8772\n",
      "Epoch 28/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3200 - acc: 0.8619 - val_loss: 0.2961 - val_acc: 0.8769\n",
      "Epoch 29/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3187 - acc: 0.8601 - val_loss: 0.3220 - val_acc: 0.8677\n",
      "Epoch 30/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3215 - acc: 0.8612 - val_loss: 0.2882 - val_acc: 0.8794\n",
      "Epoch 31/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3180 - acc: 0.8619 - val_loss: 0.2911 - val_acc: 0.8770\n",
      "Epoch 32/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3155 - acc: 0.8644 - val_loss: 0.2963 - val_acc: 0.8805\n",
      "Epoch 33/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3100 - acc: 0.8673 - val_loss: 0.2965 - val_acc: 0.8771\n",
      "Epoch 34/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3068 - acc: 0.8680 - val_loss: 0.3184 - val_acc: 0.8657\n",
      "Epoch 35/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3145 - acc: 0.8625 - val_loss: 0.2863 - val_acc: 0.8784\n",
      "Epoch 36/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3056 - acc: 0.8685 - val_loss: 0.3009 - val_acc: 0.8760\n",
      "Epoch 37/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3032 - acc: 0.8716 - val_loss: 0.3098 - val_acc: 0.8746\n",
      "Epoch 38/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3040 - acc: 0.8682 - val_loss: 0.3012 - val_acc: 0.8742\n",
      "Epoch 39/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2999 - acc: 0.8707 - val_loss: 0.3039 - val_acc: 0.8797\n",
      "Epoch 40/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.3000 - acc: 0.8732 - val_loss: 0.3064 - val_acc: 0.8786\n",
      "Epoch 41/200\n",
      "14848/15000 [============================>.] - ETA: 0s - loss: 0.3019 - acc: 0.8712\n",
      "Epoch 00040: reducing learning rate to 0.030000000447034835.\n",
      "15000/15000 [==============================] - 46s - loss: 0.3013 - acc: 0.8716 - val_loss: 0.2969 - val_acc: 0.8732\n",
      "Epoch 42/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2969 - acc: 0.8725 - val_loss: 0.2819 - val_acc: 0.8820\n",
      "Epoch 43/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2915 - acc: 0.8721 - val_loss: 0.2845 - val_acc: 0.8820\n",
      "Epoch 44/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2861 - acc: 0.8779 - val_loss: 0.2825 - val_acc: 0.8836\n",
      "Epoch 45/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2883 - acc: 0.8757 - val_loss: 0.2880 - val_acc: 0.8822\n",
      "Epoch 46/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2858 - acc: 0.8777 - val_loss: 0.2810 - val_acc: 0.8823\n",
      "Epoch 47/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2824 - acc: 0.8811 - val_loss: 0.2799 - val_acc: 0.8840\n",
      "Epoch 48/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2850 - acc: 0.8805 - val_loss: 0.2929 - val_acc: 0.8815\n",
      "Epoch 49/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2806 - acc: 0.8817 - val_loss: 0.2811 - val_acc: 0.8821\n",
      "Epoch 50/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2822 - acc: 0.8821 - val_loss: 0.2768 - val_acc: 0.8844\n",
      "Epoch 51/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2786 - acc: 0.8829 - val_loss: 0.2796 - val_acc: 0.8844\n",
      "Epoch 52/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2730 - acc: 0.8809 - val_loss: 0.2802 - val_acc: 0.8859\n",
      "Epoch 53/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2767 - acc: 0.8824 - val_loss: 0.2783 - val_acc: 0.8843\n",
      "Epoch 54/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2815 - acc: 0.8777 - val_loss: 0.2851 - val_acc: 0.8851\n",
      "Epoch 55/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2779 - acc: 0.8813 - val_loss: 0.2842 - val_acc: 0.8797\n",
      "Epoch 56/200\n",
      "14848/15000 [============================>.] - ETA: 0s - loss: 0.2740 - acc: 0.8840\n",
      "Epoch 00055: reducing learning rate to 0.009000000357627868.\n",
      "15000/15000 [==============================] - 45s - loss: 0.2741 - acc: 0.8840 - val_loss: 0.2811 - val_acc: 0.8863\n",
      "Epoch 57/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2734 - acc: 0.8834 - val_loss: 0.2798 - val_acc: 0.8853\n",
      "Epoch 58/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2775 - acc: 0.8816 - val_loss: 0.2787 - val_acc: 0.8857\n",
      "Epoch 59/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2712 - acc: 0.8835 - val_loss: 0.2786 - val_acc: 0.8849\n",
      "Epoch 60/200\n",
      "15000/15000 [==============================] - 45s - loss: 0.2705 - acc: 0.8873 - val_loss: 0.2807 - val_acc: 0.8847\n",
      "Epoch 61/200\n",
      "14848/15000 [============================>.] - ETA: 0s - loss: 0.2666 - acc: 0.8871\n",
      "Epoch 00060: reducing learning rate to 0.002700000163167715.\n",
      "15000/15000 [==============================] - 45s - loss: 0.2666 - acc: 0.8873 - val_loss: 0.2811 - val_acc: 0.8848\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6785527fd0>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([data_1[10000:],np.array(data[extra_variable][10000:])], pred[10000:],\n",
    "          batch_size=256,\n",
    "          epochs=200,\n",
    "          verbose=1,\n",
    "          validation_data=([data_1[:10000],np.array(data[extra_variable][:10000])],  pred[:10000]), \n",
    "          callbacks = [\n",
    "                      EarlyStopping(monitor='val_loss',patience=10)\n",
    "                      ,ReduceLROnPlateau(monitor='val_loss',patience=5,factor=0.3, min_lr=1e-6, verbose=1)\n",
    "                        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CONCLUSION\n",
    "\n",
    "I used unlabeledTrainData and TestData as corpus to train word2vec. In my experience it's ok. It doesn't generate overfitting. \n",
    "\n",
    "You can also decide to use the pre-trained model by Google (I've showed this model here: https://lnkd.in/dddZpKt). I preferred to train a new model because I preferred a lower-dimensional model. \n",
    "\n",
    "If you want to improve this notebook you have multiple ways:\n",
    "   \n",
    "Preprocessing:\n",
    "\n",
    "    1) You can choose better lists of words for \"positive\" and \"negative\"\n",
    "    2) You can cut by hand(!!!) some neutral most-common words. Be careful to not delete positive or negative words as \"good\",\"bad\" or similar!\n",
    "    3) You can create much more extra-variables\n",
    "    4) You can train word2vec more efficiently by tuning some parameters\n",
    "\n",
    "Model:\n",
    "\n",
    "    You have a large amount of ways to do better. I'll show you some alternatives in the future.\n",
    "    \n",
    "###### P.S. This notebook is just an example. \n",
    "In this task there are several problems. Because of the length of the comments this task looks like a document classification. \n",
    "Because of that my personal advice is straightforward in this direction: http://aclweb.org/anthology/D15-1167\n",
    "\n",
    "\n",
    "\n",
    "I'll really appreciate to be contacted for new points of view or advices as well as better solutions.\n",
    "    \n",
    "#### I hope you appreciate these examples. See you around,\n",
    "#### Federico"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}