lectures/movie_ratings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie ratings\n",
    "\n",
    "The idea for this example comes from this post: http://www.gregreda.com/2013/10/26/using-pandas-on-the-movielens-dataset/.\n",
    "\n",
    "The data can be found here: http://grouplens.org/datasets/movielens/.\n",
    "\n",
    "Documentation on the Blaze library can be found here: http://blaze.pydata.org/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pylab as plt\n",
    "import seaborn as sns\n",
    "import dask.dataframe as dd\n",
    "\n",
    "from dask.diagnostics import ProgressBar\n",
    "from odo import odo\n",
    "\n",
    "sns.set_context('notebook')\n",
    "pd.set_option('float_format', '{:6.2f}'.format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load ratings into Blaza Data object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed |  0.1s\n",
      "   userId  movieId  rating   timestamp\n",
      "0       1       31    2.50  1260759144\n",
      "1       1     1029    3.00  1260759179\n",
      "2       1     1061    3.00  1260759182\n",
      "3       1     1129    2.00  1260759185\n",
      "4       1     1172    4.00  1260759205\n"
     ]
    }
   ],
   "source": [
    "path = '/home/khrapov/ml-latest-small/'\n",
    "\n",
    "ratings = dd.read_csv(path + 'ratings.csv')\n",
    "\n",
    "with ProgressBar():\n",
    "    print(ratings.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast to Pandas, you can not just print the result. It has to be computed first. Blaze actually stores only the algorithm of operations. You have to start them by invoking either ``bz.compute`` function or convert the result to Pandas DataFrame with ``bz.odo``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Count method:\n",
      " dd.Series<datafra..., npartitions=1>\n",
      "[########################################] | 100% Completed |  0.1s\n",
      "Compute method:\n",
      " userId       100004\n",
      "movieId      100004\n",
      "rating       100004\n",
      "timestamp    100004\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "with ProgressBar():\n",
    "    print('Count method:\\n', ratings.count())\n",
    "    print('Compute method:\\n', ratings.count().compute())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read movie titles file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                      title  \\\n",
      "movieId                                       \n",
      "1                          Toy Story (1995)   \n",
      "2                            Jumanji (1995)   \n",
      "3                   Grumpier Old Men (1995)   \n",
      "4                  Waiting to Exhale (1995)   \n",
      "5        Father of the Bride Part II (1995)   \n",
      "\n",
      "                                              genres  \n",
      "movieId                                               \n",
      "1        Adventure|Animation|Children|Comedy|Fantasy  \n",
      "2                         Adventure|Children|Fantasy  \n",
      "3                                     Comedy|Romance  \n",
      "4                               Comedy|Drama|Romance  \n",
      "5                                             Comedy  \n"
     ]
    }
   ],
   "source": [
    "movies = pd.read_csv(path + 'movies.csv', index_col='movieId')\n",
    "\n",
    "print(movies.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all genres for each movie and place them on separate lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Adventure' 'Animation' 'Children' 'Comedy' 'Fantasy' 'Romance' 'Drama'\n",
      " 'Action' 'Crime' 'Thriller' 'Horror' 'Mystery' 'Sci-Fi' 'Documentary'\n",
      " 'IMAX' 'War' 'Musical' 'Western' 'Film-Noir' '(no genres listed)']\n",
      "movieId\n",
      "1    Adventure\n",
      "1    Animation\n",
      "1     Children\n",
      "1       Comedy\n",
      "1      Fantasy\n",
      "2    Adventure\n",
      "2     Children\n",
      "2      Fantasy\n",
      "3       Comedy\n",
      "3      Romance\n",
      "4       Comedy\n",
      "4        Drama\n",
      "4      Romance\n",
      "5       Comedy\n",
      "6       Action\n",
      "Name: genre, dtype: object\n"
     ]
    }
   ],
   "source": [
    "genres = movies['genres'].str.split('|').apply(pd.Series, 1).stack()\n",
    "genres.index = genres.index.droplevel(-1)\n",
    "genres.name = 'genre'\n",
    "\n",
    "print(genres.unique())\n",
    "print(genres.head(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Separate movie titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId\n",
      "1                      Toy Story (1995)\n",
      "2                        Jumanji (1995)\n",
      "3               Grumpier Old Men (1995)\n",
      "4              Waiting to Exhale (1995)\n",
      "5    Father of the Bride Part II (1995)\n",
      "Name: title, dtype: object\n"
     ]
    }
   ],
   "source": [
    "movie_titles = movies.loc[:, 'title']\n",
    "\n",
    "print(movie_titles.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the most rated movies?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed |  0.2s\n",
      "title\n",
      "Forrest Gump (1994)                          341\n",
      "Pulp Fiction (1994)                          324\n",
      "Shawshank Redemption, The (1994)             311\n",
      "Silence of the Lambs, The (1991)             304\n",
      "Star Wars: Episode IV - A New Hope (1977)    291\n",
      "Name: rating, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "most_rated = ratings.groupby('movieId').count()\n",
    "most_rated = most_rated.join(odo(movie_titles, dd.DataFrame, npartitions=1))\n",
    "most_rated = most_rated.set_index('title')\n",
    "df = most_rated.nlargest(n=5, columns=['rating'])\n",
    "\n",
    "with ProgressBar():\n",
    "    print(df.compute()['rating'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the highest rated movies with minimum number of rating counts?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed |  4.9s\n",
      "                                  count  mean_rating\n",
      "title                                               \n",
      "Inherit the Wind (1960)           12.00         4.54\n",
      "Godfather, The (1972)            200.00         4.49\n",
      "Shawshank Redemption, The (1994) 311.00         4.49\n",
      "Tom Jones (1963)                  12.00         4.46\n",
      "On the Waterfront (1954)          29.00         4.45\n",
      "                                           count  mean_rating\n",
      "title                                                        \n",
      "Battlefield Earth (2000)                   19.00         1.21\n",
      "Speed 2: Cruise Control (1997)             23.00         1.65\n",
      "Police Academy 6: City Under Siege (1989)  12.00         1.71\n",
      "Super Mario Bros. (1993)                   17.00         1.74\n",
      "Blade: Trinity (2004)                      12.00         1.79\n"
     ]
    }
   ],
   "source": [
    "grouped = ratings.groupby('movieId')[['rating']]\n",
    "\n",
    "def summary(series):\n",
    "    series = series['rating']\n",
    "    return pd.Series({'mean_rating': series.mean(), 'count': series.count()})\n",
    "\n",
    "movie_stats = grouped.apply(summary, meta={'mean_rating': 'f8', 'count': 'i4'})\n",
    "movie_stats = movie_stats[movie_stats['count'] > movie_stats['count'].mean()]\n",
    "movie_stats = movie_stats.join(odo(movie_titles, dd.DataFrame, npartitions=1))\n",
    "movie_stats = movie_stats.set_index('title')\n",
    "df = movie_stats.nlargest(n=5, columns=['mean_rating'])\n",
    "\n",
    "with ProgressBar():\n",
    "    print(df.compute())\n",
    "\n",
    "df = odo(movie_stats, pd.DataFrame).nsmallest(n=5, columns=['mean_rating'])\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract year from movie title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                     title  year\n",
      "movieId                                                         \n",
      "164979                               Women of '69, Unboxed  -999\n",
      "162376                                     Stranger Things  -999\n",
      "151307                           The Lovers and the Despot  -999\n",
      "143410                                          Hyena Road  -999\n",
      "32898    Trip to the Moon, A (Voyage dans la lune, Le) ...  1902\n",
      "title    object\n",
      "year      int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "df = pd.DataFrame(movie_titles)\n",
    "\n",
    "def find_year(x):\n",
    "    try:\n",
    "        return int(re.findall('\\((\\d+)', x)[-1])\n",
    "    except:\n",
    "        return -999\n",
    "    \n",
    "df['year'] = df['title'].apply(find_year)\n",
    "df.sort_values(by='year', inplace=True)\n",
    "\n",
    "print(df.head())\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the highest rated movies by genre?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed |  8.1s\n",
      "[########################################] | 100% Completed |  7.6s\n",
      "                                            title  mean_rating\n",
      "2603                      Schindler's List (1993)         5.00\n",
      "2606            Terminator 2: Judgment Day (1991)         5.00\n",
      "2610                    African Queen, The (1951)         5.00\n",
      "2614  Wallace & Gromit: The Wrong Trousers (1993)         5.00\n",
      "2624     Treasure of the Sierra Madre, The (1948)         5.00\n"
     ]
    }
   ],
   "source": [
    "df = ratings.join(odo(genres, dd.DataFrame, npartitions=1))\n",
    "\n",
    "grouped = df.groupby(['genre', 'movieId'])[['rating']]\n",
    "\n",
    "def summary(series):\n",
    "    series = series['rating']\n",
    "    return pd.Series({'mean_rating': series.mean(), 'count': series.count()})\n",
    "\n",
    "movie_stats = grouped.apply(summary, meta={'mean_rating': 'f8', 'count': 'i4'})\n",
    "movie_stats = movie_stats[movie_stats['count'] > movie_stats['count'].mean()]\n",
    "movie_stats = movie_stats.join(odo(movie_titles, dd.DataFrame, npartitions=1))\n",
    "with ProgressBar():\n",
    "    movie_stats = odo(movie_stats, pd.DataFrame)\n",
    "\n",
    "df = movie_stats.reset_index().sort_values(by=['genre', 'mean_rating'], ascending=False)\n",
    "df = df.groupby('genre')[['title', 'mean_rating']].apply(lambda x: x.head())\n",
    "\n",
    "print(df.ix['Sci-Fi'])"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}