04. GaussianNB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://www.upm.es/sfs/Rectorado/Gabinete%20del%20Rector/Logos/UPM/EscPolitecnica/EscUpmPolit_p.gif \"UPM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trabajo final SITC\n",
    "## Análisis de sentimientos en Twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Departamento de Ingeniería de Sistemas Telemáticos. Universidad Politécnica de Madrid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Realizado por:\n",
    "- Juan Bermudo Mera\n",
    "- Margarita Bolívar Jiménez\n",
    "- Lourdes Fernández Nieto\n",
    "- Ramón Pérez Hernández\n",
    "\n",
    "© 2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Algoritmo GaussianNB aplicado sobre el fichero de tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tabla de contenidos\n",
    "\n",
    "* [Importación de datos necesarios para aplicar el algoritmo](#1.-Importación-de-datos-necesarios-para-aplicar-el-algoritmo)\n",
    "\t* [Importación de librerías](#Importación-de-librerías)\n",
    "    * [Importación de corpus y tweets](#Importación-de-corpus-y-tweets)\n",
    "    * [Tokenización y stemming](#Tokenización-y-stemming)\n",
    "* [Entrenamiento del modelo](#2.-Entrenamiento-del-modelo)\n",
    "* [Rendimiento del modelo](#3.-Rendimiento-del-modelo)\n",
    "* [Predicción de la polaridad](#4.-Predicción-de-la-polaridad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Importación de datos necesarios para aplicar el algoritmo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Importación de librerías"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Lourdes/anaconda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "/Users/Lourdes/anaconda/lib/python3.6/site-packages/sklearn/grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Importamos librerías. Las que no están instaladas, instalar con pip install <nombre_paquete>\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import nltk\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "from string import punctuation\n",
    "from nltk.stem import SnowballStemmer\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from mlxtend.preprocessing import DenseTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Importación de corpus y tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Se importan el corpus y los tweets\n",
    "tweets_df = pd.read_excel('ficheros/Preprocesados/tweets_corpus_header.xlsx', header=0, encoding='iso8859_15')\n",
    "tweets = pd.read_excel('ficheros/TweetsConTopic/tweets_spainGeo_topic.xlsx', header=0, encoding='iso8859_15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Tokenización y stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/Lourdes/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Se descargan las palabras de parada en español\n",
    "nltk.download(\"stopwords\")\n",
    "spanish_stopwords = stopwords.words('spanish')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Obtenemos los signos de puntuación que se utilizan en español\n",
    "non_words = list(punctuation)\n",
    "non_words.extend(['¿', '¡'])\n",
    "non_words.extend(map(str,range(10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Se definen las funciones para realizar la tokenización y el stemming\n",
    "stemmer = SnowballStemmer('spanish')\n",
    "tknzr = TweetTokenizer(strip_handles=True, reduce_len=True)\n",
    "\n",
    "def stem_tokens(tokens, stemmer):\n",
    "    stemmed = []\n",
    "    for item in tokens:\n",
    "        stemmed.append(stemmer.stem(item))\n",
    "    return stemmed\n",
    "\n",
    "def tokenize(text):\n",
    "    # Eliminamos lo que no sean palabras\n",
    "    text = ''.join([c for c in text if c not in non_words])\n",
    "    # Tokenización\n",
    "    tokens = tknzr.tokenize(text)\n",
    "\n",
    "    # Stemming\n",
    "    try:\n",
    "        stems = stem_tokens(tokens, stemmer)\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        print(text)\n",
    "        stems = ['']\n",
    "    return stems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Entrenamiento del modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'priors': None}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Buscamos los parámetros que podemos utilizar para entrenar el modelo\n",
    "GaussianNB().get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El algortimo GaussianNB, tiene un parámetro denominado priors. Este parámetro especifica las probabilidades previas de las clases. En nuestro caso, no queremos especificarlas porque queremos que las probabilidades se ajusten según los datos --> No hace falta que busquemos cuáles son los mejores parámetros que se ajusten al modelo --> No realizamos la fase de entrenamiento y pasamos directamente a conocer cuál es el rendimiento del modelo y a realizar la predicción de polaridad de los tweets con este modelo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "## 3. Rendimiento del modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.79556041006056721"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mediante validación cruzada obtenemos el rendimiento del modelo\n",
    "model = GaussianNB()\n",
    "vectorizer = CountVectorizer(\n",
    "    analyzer = 'word',\n",
    "    tokenizer = tokenize,\n",
    "    lowercase = True,\n",
    "    stop_words = spanish_stopwords,\n",
    "    min_df = 0,\n",
    "    max_df = 4700,\n",
    "    max_features=1000\n",
    ")\n",
    "\n",
    "tweets_df_data_features = vectorizer.fit_transform(tweets_df.content)\n",
    "tweets_df_data_features_nd = tweets_df_data_features.toarray()\n",
    "\n",
    "scores = cross_val_score(\n",
    "    model,\n",
    "    tweets_df_data_features_nd[0:len(tweets_df)],\n",
    "    y=tweets_df.polarity_bin,\n",
    "    scoring='roc_auc',\n",
    "    cv=None\n",
    "    )\n",
    "\n",
    "scores.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "El valor que se obtiene del rendimiento del modelo para la métrica de Área bajo la Curva ROC es de 0.79556041006056721"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4. Predicción de la polaridad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Una vez que tenemos el modelo que mejor métrica nos aporta (tras realizar muchas pruebas con distintas métricas \n",
    "# y parámetros pasados al modelo), volvemos a crear un pipeline pero en este caso, pasándole los mejores parámetros\n",
    "# obtenidos del SVC para predecir qué polaridad tienen los tweets que están aún sin etiquetar\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer(\n",
    "            analyzer = 'word',\n",
    "            tokenizer = tokenize,\n",
    "            lowercase = True,\n",
    "            stop_words = spanish_stopwords,\n",
    "            min_df = 0,\n",
    "            max_df = 26363,\n",
    "            max_features=1000\n",
    "            )),\n",
    "    ('to_dense', DenseTransformer()),\n",
    "    ('cls', GaussianNB())\n",
    "])\n",
    "\n",
    "pipeline.fit(tweets_df.content, tweets_df.polarity_bin)\n",
    "tweets['polarity'] = pipeline.predict(tweets.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>polarity</th>\n",
       "      <th>Topic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6175</th>\n",
       "      <td>oferta de #empleo para técnico de iluminación ...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17211</th>\n",
       "      <td>Te miro y te juro que entre hoja y hoja se te ...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1231</th>\n",
       "      <td>¿Tienes una idea más clara de lo que quieres h...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>568</th>\n",
       "      <td>illo que ya mismo empieza juego de tronossssss...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14449</th>\n",
       "      <td>que bonito eso de tenerte ¿no? te quiero @ Alc...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8971</th>\n",
       "      <td>Artículos más comentados \\n\\n…\\n\\nLeer mas en:...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2060</th>\n",
       "      <td>@Angel_Ortiz_G @laSextaTV @_anapastor_ Sigue l...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11254</th>\n",
       "      <td>Cristóbal Rosaleny, @crosaleny es ahora una te...</td>\n",
       "      <td>0</td>\n",
       "      <td>tendencias-twitter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19951</th>\n",
       "      <td>Si te gusta la tortilla de patatas y el pulpo ...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13045</th>\n",
       "      <td>Sooy el boss @ Estación de Rivas-Urbanizaciones</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14687</th>\n",
       "      <td>@HoroscopoViral Por supuesto que sí!</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1930</th>\n",
       "      <td>Cuba: no retornar al capitalismo jamás</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3067</th>\n",
       "      <td>Iglesias ataca a la prensa</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8457</th>\n",
       "      <td>LG G5 review (en español), Smartphone con dobl...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14832</th>\n",
       "      <td>#RETENCIÓN nivel AMARILLO en AUTOPISTA / AUTOV...</td>\n",
       "      <td>1</td>\n",
       "      <td>trafico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14531</th>\n",
       "      <td>Un &amp;lt; y un 3 @ Malasaña</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22868</th>\n",
       "      <td>Buena fiesta con gente \\nGracias por una gran ...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6873</th>\n",
       "      <td>Algunas respuestas a por qué el diseño instruc...</td>\n",
       "      <td>0</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26040</th>\n",
       "      <td>Dando cela y puliendo cela de camino a Valenci...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15108</th>\n",
       "      <td>@pnique Jomío tú estás pa dar gracias a España...</td>\n",
       "      <td>1</td>\n",
       "      <td>otros</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 content  polarity  \\\n",
       "6175   oferta de #empleo para técnico de iluminación ...         0   \n",
       "17211  Te miro y te juro que entre hoja y hoja se te ...         1   \n",
       "1231   ¿Tienes una idea más clara de lo que quieres h...         1   \n",
       "568    illo que ya mismo empieza juego de tronossssss...         0   \n",
       "14449  que bonito eso de tenerte ¿no? te quiero @ Alc...         1   \n",
       "8971   Artículos más comentados \\n\\n…\\n\\nLeer mas en:...         1   \n",
       "2060   @Angel_Ortiz_G @laSextaTV @_anapastor_ Sigue l...         0   \n",
       "11254  Cristóbal Rosaleny, @crosaleny es ahora una te...         0   \n",
       "19951  Si te gusta la tortilla de patatas y el pulpo ...         0   \n",
       "13045   Sooy el boss @ Estación de Rivas-Urbanizaciones          0   \n",
       "14687               @HoroscopoViral Por supuesto que sí!         1   \n",
       "1930             Cuba: no retornar al capitalismo jamás          0   \n",
       "3067                         Iglesias ataca a la prensa          0   \n",
       "8457   LG G5 review (en español), Smartphone con dobl...         0   \n",
       "14832  #RETENCIÓN nivel AMARILLO en AUTOPISTA / AUTOV...         1   \n",
       "14531                         Un &lt; y un 3 @ Malasaña          0   \n",
       "22868  Buena fiesta con gente \\nGracias por una gran ...         1   \n",
       "6873   Algunas respuestas a por qué el diseño instruc...         0   \n",
       "26040  Dando cela y puliendo cela de camino a Valenci...         1   \n",
       "15108  @pnique Jomío tú estás pa dar gracias a España...         1   \n",
       "\n",
       "                    Topic  \n",
       "6175                otros  \n",
       "17211               otros  \n",
       "1231                otros  \n",
       "568                 otros  \n",
       "14449               otros  \n",
       "8971                otros  \n",
       "2060                otros  \n",
       "11254  tendencias-twitter  \n",
       "19951               otros  \n",
       "13045               otros  \n",
       "14687               otros  \n",
       "1930                otros  \n",
       "3067                otros  \n",
       "8457                otros  \n",
       "14832             trafico  \n",
       "14531               otros  \n",
       "22868               otros  \n",
       "6873                otros  \n",
       "26040               otros  \n",
       "15108               otros  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mostramos algunos de los tweets que han sido etiquetados con la polaridad\n",
    "tweets[['content', 'polarity','Topic']].sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Guardamos los tweets en un fichero csv con su polaridad\n",
    "tweets[[ 'content', 'Latitude', 'Longitude', 'polarity', 'Topic']].to_csv('ficheros/TweetsConPolaridadYTopic/tweetsGaussianNB_polarity_bin.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Guardamos los tweets en fichero excel\n",
    "tweets[['content','Latitude','Longitude','polarity','Topic']].to_excel('ficheros/TweetsConPolaridadYTopicExcel/tweetsGaussianNB_polarity_bin.xlsx', header=True, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Licencia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El notebook está licenciado libremente bajo la licencia [Creative Commons Attribution Share-Alike](https://creativecommons.org/licenses/by/2.0/).\n",
    "\n",
    "La base del código empleado procede del trabajo de Manuel Garrido llamado [Cómo hacer Análisis de Sentimiento en español](http://pybonacci.org/2015/11/24/como-hacer-analisis-de-sentimiento-en-espanol-2/).\n",
    "\n",
    "© 2017 - Juan Bermudo Mera, Margarita Bolívar Jiménez, Lourdes Fernández Nieto, Ramón Pérez Hernández.\n",
    "\n",
    "Universidad Politécnica de Madrid."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}