pre-trained-embeddings/Pre-trained embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using pre-trained embeddings and NLP corpora\n",
    "\n",
    "Gensim has some really nice functionality, in that it allows you to use pre-trained GloVe and Word2Vec embeddings with its libraries. In addition there are also some re-usable corpora that you can download and immediately use to train a Word2Vec embedding. The code snippets below show you how. The source of the embeddings can be found here: https://github.com/RaRe-Technologies/gensim-data.\n",
    "\n",
    "I'll have to warn you that I'm not impressed with the quality of the pre-trained word embeddings. Either the dataset is noisy or its just too general. To be explained more later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.word2vec import Word2Vec\n",
    "import gensim.downloader as api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained: Twitter GloVe Embeddings\n",
    "This first step downloads the pre-trained embeddings and loads it for re-use. Note that these are GloVe embeddings built using Tweets as the name suggests. These vectors are based on 2B tweets, 27B tokens, 1.2M vocab, uncased. The original source can be found here: https://nlp.stanford.edu/projects/glove/. The `25` in the model name refers to the dimensionality of the vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the model and return as object ready for use\n",
    "model_glove_twitter = api.load(\"glove-twitter-25\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have loaded the pre-trained model, just use it as you would with any gensim word2vec model. Here are a few similarity examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('clegg', 0.9653651118278503),\n",
       " ('miliband', 0.9515050053596497),\n",
       " ('bachmann', 0.9484400749206543),\n",
       " ('mcconnell', 0.9416399002075195),\n",
       " ('carney', 0.934025764465332),\n",
       " ('coulter', 0.9311323761940002),\n",
       " ('boehner', 0.9286302328109741),\n",
       " ('santorum', 0.9269059300422668),\n",
       " ('farage', 0.919365406036377),\n",
       " ('mourdock', 0.9186689853668213)]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_glove_twitter.wv.most_similar(\"pelosi\",topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('policy', 0.9484813213348389),\n",
       " ('reforms', 0.9403933882713318),\n",
       " ('laws', 0.94012051820755),\n",
       " ('government', 0.923071026802063),\n",
       " ('regulations', 0.916893482208252),\n",
       " ('economy', 0.9110006093978882),\n",
       " ('immigration', 0.9105910062789917),\n",
       " ('legislation', 0.908964991569519),\n",
       " ('govt', 0.9054746627807617),\n",
       " ('regulation', 0.9050779342651367)]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_glove_twitter.wv.most_similar(\"policies\",topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which of these words don't fit?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'orange'"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#what doesn't fit?\n",
    "model_glove_twitter.wv.doesnt_match([\"trump\",\"bernie\",\"obama\",\"pelosi\",\"orange\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word vectors for `trump` and `obama`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([-0.56174 ,  0.69419 ,  0.16733 ,  0.055867, -0.26266 , -0.6303  ,\n",
       "        -0.28311 , -0.88244 ,  0.57317 , -0.82376 ,  0.46728 ,  0.48607 ,\n",
       "        -2.1942  , -0.41972 ,  0.31795 , -0.70063 ,  0.060693,  0.45279 ,\n",
       "         0.6564  ,  0.20738 ,  0.84496 , -0.087537, -0.38856 , -0.97028 ,\n",
       "        -0.40427 ], dtype=float32),\n",
       " array([ 0.77126 ,  0.81259 , -0.5901  , -0.015908, -0.082797, -1.2261  ,\n",
       "         0.098286,  0.087488,  0.012586, -0.35884 ,  0.80733 ,  0.12569 ,\n",
       "        -4.0522  ,  0.14856 ,  0.6988  , -0.78948 , -0.77125 ,  0.49512 ,\n",
       "         0.16366 , -0.9713  ,  0.95064 ,  0.19921 , -0.27903 , -1.6844  ,\n",
       "        -0.79424 ], dtype=float32))"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show weight vector for trump and obama\n",
    "model_glove_twitter[\"trump\"],model_glove_twitter['obama']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank phrases by similarity\n",
    "\n",
    "The goal here is given a query phrase, rank all other phrases by semantic similarity (using the glove twitter embeddings) and compare that with surface level similarity using jaccard similarity index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrases most similar to 'barack hussain obama' using glove word embeddings\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>phrase</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>barrack obama</td>\n",
       "      <td>0.956801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>barrack h. obama</td>\n",
       "      <td>0.944671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>barrack hussein obama</td>\n",
       "      <td>0.937000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>michelle obama</td>\n",
       "      <td>0.905201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>donald trump</td>\n",
       "      <td>0.729601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>melania trump</td>\n",
       "      <td>0.614963</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  phrase     score\n",
       "0          barrack obama  0.956801\n",
       "1       barrack h. obama  0.944671\n",
       "2  barrack hussein obama  0.937000\n",
       "3         michelle obama  0.905201\n",
       "4           donald trump  0.729601\n",
       "5          melania trump  0.614963"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.metrics import jaccard_similarity_score\n",
    "\n",
    "phrases=[\"barrack obama\",\"barrack h. obama\",\"barrack hussein obama\",\"michelle obama\",\"donald trump\",\"melania trump\"]\n",
    "query=\"barack hussain obama\"\n",
    "\n",
    "results_glove=[]\n",
    "results_jaccard=[]\n",
    "\n",
    "def compute_jaccard(t1,t2):\n",
    "    \n",
    "    intersect = [value for value in t1 if value in t2] \n",
    "    \n",
    "    union=[]\n",
    "    union.extend(t1)\n",
    "    union.extend(t2)\n",
    "    union=list(set(union))\n",
    "    \n",
    "    \n",
    "    jaccard=(len(intersect))/(len(union)+0.01)\n",
    "    return jaccard\n",
    "    \n",
    "\n",
    "for p in phrases:\n",
    "    tokens_1=[t for t in p.split() if t in model.wv.vocab]\n",
    "    tokens_2=[t for t in query.split() if t in model.wv.vocab]\n",
    "    \n",
    "    #compute jaccard similarity\n",
    "    jaccard=compute_jaccard(tokens_1,tokens_2)\n",
    "    results_jaccard.append([p,jaccard])\n",
    "    \n",
    "    #compute cosine similarity using word embedings \n",
    "    cosine=0\n",
    "    if (len(tokens_1) > 0 and len(tokens_2)>0):\n",
    "        cosine=model_glove_twitter.wv.n_similarity(tokens_1,tokens_2)\n",
    "        results_glove.append([p,cosine])\n",
    "\n",
    "print(\"Phrases most similar to '{0}' using glove word embeddings\".format(query))\n",
    "pd.DataFrame(results_glove,columns=[\"phrase\",\"score\"]).sort_values(by=[\"score\"],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrases most similar to 'barack hussain obama' using jaccard similarity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>phrase</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>barrack obama</td>\n",
       "      <td>0.249377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>michelle obama</td>\n",
       "      <td>0.249377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>barrack h. obama</td>\n",
       "      <td>0.199601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>barrack hussein obama</td>\n",
       "      <td>0.199601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>donald trump</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>melania trump</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  phrase     score\n",
       "0          barrack obama  0.249377\n",
       "3         michelle obama  0.249377\n",
       "1       barrack h. obama  0.199601\n",
       "2  barrack hussein obama  0.199601\n",
       "4           donald trump  0.000000\n",
       "5          melania trump  0.000000"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Phrases most similar to '{0}' using jaccard similarity\".format(query))\n",
    "pd.DataFrame(results_jaccard,columns=[\"phrase\",\"score\"]).sort_values(by=[\"score\"],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-trainend: GloVe Wikipedia + Gigaword \n",
    "The example below uses pre-trained GloVe vectors based on Wikipedia 2014 and Gigaword. The original source of these embeddings can be found here: https://nlp.stanford.edu/projects/glove/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[==================================================] 100.0% 128.1/128.1MB downloaded\n"
     ]
    }
   ],
   "source": [
    "#again, download and load the model\n",
    "model_gigaword = api.load(\"glove-wiki-gigaword-100\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('filthy', 0.7690386176109314),\n",
       " ('smelly', 0.7392697334289551),\n",
       " ('shabby', 0.7025482654571533),\n",
       " ('dingy', 0.7022336721420288),\n",
       " ('grubby', 0.675451397895813),\n",
       " ('grungy', 0.6414023637771606),\n",
       " ('dank', 0.626369833946228),\n",
       " ('sweaty', 0.622745156288147),\n",
       " ('dreary', 0.6216243505477905),\n",
       " ('gritty', 0.621574878692627)]"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find similarity\n",
    "model_gigaword.wv.most_similar(positive=['dirty','grimy'],topn=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('spring', 0.8519278764724731),\n",
       " ('autumn', 0.7865706086158752),\n",
       " ('olympics', 0.6915044784545898),\n",
       " ('weekend', 0.6908973455429077),\n",
       " ('days', 0.6872981786727905),\n",
       " ('during', 0.6861997842788696),\n",
       " ('season', 0.6849778890609741),\n",
       " ('year', 0.6827663779258728),\n",
       " ('rainy', 0.6744829416275024),\n",
       " ('day', 0.671191930770874)]"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_gigaword.wv.most_similar(positive=[\"summer\",\"winter\"],topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load a dataset and train a model\n",
    "Instead of loading pre-trained embeddings, you can also load a corpus and train it on demand. This list of datasets that you can download can be found here: https://github.com/RaRe-Technologies/gensim-data#datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[==================================================] 100.0% 31.6/31.6MB downloaded\n"
     ]
    }
   ],
   "source": [
    "from gensim.models.word2vec import Word2Vec\n",
    "\n",
    "# this loads the text8 dataset\n",
    "corpus = api.load('text8')\n",
    "\n",
    "# train a Word2Vec model\n",
    "model_text8 = Word2Vec(corpus,iter=10,size=150, window=10, min_count=2, workers=10)  # train a model from the corpus\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('outraged', 0.7200734615325928),\n",
       " ('surprised', 0.6967819333076477),\n",
       " ('greeted', 0.6692871451377869),\n",
       " ('angered', 0.6468496322631836),\n",
       " ('confronted', 0.6217055320739746),\n",
       " ('beaten', 0.6206371188163757),\n",
       " ('betrayed', 0.6194607019424438),\n",
       " ('disgusted', 0.6146512031555176),\n",
       " ('amused', 0.6022583842277527),\n",
       " ('offended', 0.6014840602874756)]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# similarity \n",
    "model_text8.wv.most_similar(\"shocked\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.45678782"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# similarity between two different words\n",
    "model_text8.wv.similarity(w1=\"dirty\",w2=\"smelly\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'france'"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Which one is the odd one out in this list?\n",
    "model_text8.wv.doesnt_match([\"cat\",\"dog\",\"france\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}