evaluation_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Notebook\n",
    "\n",
    "I evaluated 4 models before going with the model deployed on the web app: \n",
    "\n",
    "* Model 1: Used TextStat, Gensim, and Spacy *(omitted)*\n",
    "* Model 2: Used Latent Dirichlet Allocation (LDA) topic modeling with 10 topics, then sorts user content into a topic. \n",
    "* Model 3: Used TextStat and TF-IDF Vectorizer with 2000 dimensions. \n",
    "* Model 4: Used TextStat and TF-IDF Vectorizer with top 20 features.\n",
    "\n",
    "Each iteration was done to so the resulting content was more similar to the user input content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 2: LDA Topic Modeling \n",
    "\n",
    "Sort the corpus into 10 topics and then assign probabilities of each document belonging to each topic. Automate labeling by assigning the article with the topic that has the highest probability. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from scipy.spatial.distance import cosine, cdist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/df_corpus2.pkl\", \"rb\") as input_file:\n",
    "    df_corpus2 = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Nature Reviews Nephrology (before 2009, Nature...</td>\n",
       "      <td>20.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Food blogging represents a complex interweavin...</td>\n",
       "      <td>13.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The International Motorcycle Shows are a serie...</td>\n",
       "      <td>13.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Education in Ancient Greece was vastly \"democr...</td>\n",
       "      <td>11.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Rural economics is the study of rural economie...</td>\n",
       "      <td>12.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             content  score\n",
       "0  Nature Reviews Nephrology (before 2009, Nature...   20.6\n",
       "1  Food blogging represents a complex interweavin...   13.6\n",
       "2  The International Motorcycle Shows are a serie...   13.3\n",
       "3  Education in Ancient Greece was vastly \"democr...   11.3\n",
       "4  Rural economics is the study of rural economie...   12.5"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_corpus2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "#! pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     /Users/sherzyang/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gensim\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
    "from nltk.stem.porter import *\n",
    "import numpy as np\n",
    "np.random.seed(2018)\n",
    "import nltk\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lemmatize_stemming(text):\n",
    "    return WordNetLemmatizer().lemmatize(text, pos='v')\n",
    "def preprocess(text):\n",
    "    result = []\n",
    "    for token in gensim.utils.simple_preprocess(text):\n",
    "        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:\n",
    "            result.append(lemmatize_stemming(token))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_docs = df_corpus2['content'].map(preprocess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pickle\n",
    "# with open('data/processed_docs_lda.pkl', 'wb') as f:\n",
    "#     pickle.dump(processed_docs, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/processed_docs_lda.pkl\", \"rb\") as input_file:\n",
    "    df_corpus2 = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = gensim.corpora.Dictionary(processed_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(\"data/dictionary.pkl\", \"wb\") as f: \n",
    "#     pickle.dump(dictionary, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 adult\n",
      "1 apply\n",
      "2 cancer\n",
      "3 child\n",
      "4 chronic\n",
      "5 citation\n",
      "6 clinical\n",
      "7 coverage\n",
      "8 diagnosis\n",
      "9 dialysis\n",
      "10 disorder\n"
     ]
    }
   ],
   "source": [
    "dictionary = gensim.corpora.Dictionary(processed_docs)\n",
    "count = 0\n",
    "for k, v in dictionary.iteritems():\n",
    "    print(k, v)\n",
    "    count += 1\n",
    "    if count > 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.11463756621225898),\n",
      " (1, 0.05765586532617539),\n",
      " (2, 0.13892712182502004),\n",
      " (3, 0.09102312571345948),\n",
      " (4, 0.1579255683226295),\n",
      " (5, 0.15342950456468332),\n",
      " (6, 0.12687624510705903),\n",
      " (7, 0.13979034449587607),\n",
      " (8, 0.15698604788106313),\n",
      " (9, 0.11687015539050881),\n",
      " (10, 0.19033509306520321),\n",
      " (11, 0.0714941413459945),\n",
      " (12, 0.10949456875642996),\n",
      " (13, 0.15330801836500396),\n",
      " (14, 0.2324912851948214),\n",
      " (15, 0.2429152820827657),\n",
      " (16, 0.07326354021919622),\n",
      " (17, 0.10778082117698098),\n",
      " (18, 0.1918594902864103),\n",
      " (19, 0.24513212262921247),\n",
      " (20, 0.13877712452498228),\n",
      " (21, 0.20848419321557235),\n",
      " (22, 0.09049542779379403),\n",
      " (23, 0.27958124452493754),\n",
      " (24, 0.18772634012563447),\n",
      " (25, 0.10850653149179024),\n",
      " (26, 0.16206489016436754),\n",
      " (27, 0.05663309758786264),\n",
      " (28, 0.1465052227617137),\n",
      " (29, 0.05518954915829813),\n",
      " (30, 0.2653244279611047),\n",
      " (31, 0.11564191336679187),\n",
      " (32, 0.2139591889419809),\n",
      " (33, 0.32931380435014446),\n",
      " (34, 0.17846492643608908),\n",
      " (35, 0.09913014119814909)]\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora, models\n",
    "tfidf = models.TfidfModel(bow_corpus)\n",
    "corpus_tfidf = tfidf[bow_corpus]\n",
    "from pprint import pprint\n",
    "for doc in corpus_tfidf:\n",
    "    pprint(doc)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open (r\"data/lda_model_1.pkl\", \"rb\") as input_file:\n",
    "    lda_model = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=10, id2word=dictionary, passes=2, workers=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic: 0 \n",
      "Words: 0.007*\"century\" + 0.007*\"literature\" + 0.006*\"write\" + 0.004*\"book\" + 0.004*\"displaystyle\" + 0.004*\"bear\" + 0.003*\"ancient\" + 0.003*\"roman\" + 0.003*\"form\" + 0.003*\"museum\"\n",
      "Topic: 1 \n",
      "Words: 0.006*\"self\" + 0.006*\"music\" + 0.006*\"culture\" + 0.006*\"people\" + 0.005*\"health\" + 0.004*\"psychology\" + 0.004*\"group\" + 0.003*\"study\" + 0.003*\"social\" + 0.003*\"form\"\n",
      "Topic: 2 \n",
      "Words: 0.007*\"social\" + 0.006*\"study\" + 0.005*\"theory\" + 0.004*\"science\" + 0.004*\"human\" + 0.004*\"philosophy\" + 0.004*\"self\" + 0.004*\"research\" + 0.003*\"society\" + 0.003*\"people\"\n",
      "Topic: 3 \n",
      "Words: 0.006*\"film\" + 0.006*\"series\" + 0.005*\"star\" + 0.004*\"television\" + 0.004*\"american\" + 0.003*\"award\" + 0.003*\"music\" + 0.003*\"show\" + 0.003*\"release\" + 0.003*\"season\"\n",
      "Topic: 4 \n",
      "Words: 0.009*\"university\" + 0.007*\"program\" + 0.007*\"engineer\" + 0.005*\"school\" + 0.005*\"technology\" + 0.005*\"science\" + 0.004*\"college\" + 0.004*\"education\" + 0.004*\"research\" + 0.004*\"institute\"\n",
      "Topic: 5 \n",
      "Words: 0.008*\"food\" + 0.006*\"company\" + 0.005*\"market\" + 0.004*\"drink\" + 0.004*\"finance\" + 0.004*\"service\" + 0.003*\"world\" + 0.003*\"unite\" + 0.003*\"financial\" + 0.003*\"bank\"\n",
      "Topic: 6 \n",
      "Words: 0.004*\"government\" + 0.004*\"century\" + 0.004*\"unite\" + 0.003*\"people\" + 0.003*\"world\" + 0.003*\"city\" + 0.003*\"economic\" + 0.003*\"empire\" + 0.003*\"south\" + 0.003*\"force\"\n",
      "Topic: 7 \n",
      "Words: 0.019*\"language\" + 0.016*\"film\" + 0.007*\"languages\" + 0.005*\"write\" + 0.005*\"word\" + 0.005*\"speak\" + 0.004*\"form\" + 0.004*\"english\" + 0.003*\"century\" + 0.003*\"people\"\n",
      "Topic: 8 \n",
      "Words: 0.008*\"road\" + 0.007*\"county\" + 0.006*\"city\" + 0.005*\"highway\" + 0.004*\"north\" + 0.004*\"miles\" + 0.004*\"route\" + 0.004*\"south\" + 0.003*\"line\" + 0.003*\"east\"\n",
      "Topic: 9 \n",
      "Words: 0.016*\"music\" + 0.004*\"record\" + 0.004*\"number\" + 0.003*\"release\" + 0.003*\"group\" + 0.003*\"rock\" + 0.003*\"album\" + 0.003*\"band\" + 0.003*\"begin\" + 0.003*\"displaystyle\"\n"
     ]
    }
   ],
   "source": [
    "for idx, topic in lda_model.print_topics(-1):\n",
    "    print('Topic: {} \\nWords: {}'.format(idx, topic))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic: 0 Word: 0.002*\"film\" + 0.002*\"century\" + 0.001*\"literature\" + 0.001*\"language\" + 0.001*\"roman\" + 0.001*\"ancient\" + 0.001*\"bear\" + 0.001*\"movies\" + 0.001*\"chinese\" + 0.001*\"museum\"\n",
      "Topic: 1 Word: 0.003*\"cities\" + 0.002*\"music\" + 0.002*\"psychology\" + 0.001*\"county\" + 0.001*\"list\" + 0.001*\"city\" + 0.001*\"road\" + 0.001*\"fitness\" + 0.001*\"journal\" + 0.001*\"miles\"\n",
      "Topic: 2 Word: 0.002*\"social\" + 0.002*\"journal\" + 0.002*\"science\" + 0.002*\"psychology\" + 0.002*\"self\" + 0.001*\"philosophy\" + 0.001*\"anthropology\" + 0.001*\"study\" + 0.001*\"university\" + 0.001*\"economics\"\n",
      "Topic: 3 Word: 0.004*\"film\" + 0.003*\"music\" + 0.002*\"award\" + 0.002*\"actors\" + 0.002*\"television\" + 0.002*\"album\" + 0.001*\"star\" + 0.001*\"series\" + 0.001*\"release\" + 0.001*\"list\"\n",
      "Topic: 4 Word: 0.003*\"cities\" + 0.002*\"population\" + 0.002*\"list\" + 0.001*\"language\" + 0.001*\"census\" + 0.001*\"food\" + 0.001*\"languages\" + 0.001*\"city\" + 0.001*\"towns\" + 0.001*\"municipalities\"\n",
      "Topic: 5 Word: 0.003*\"food\" + 0.003*\"finance\" + 0.003*\"transportation\" + 0.003*\"company\" + 0.002*\"drink\" + 0.002*\"service\" + 0.002*\"market\" + 0.002*\"global\" + 0.001*\"bank\" + 0.001*\"financial\"\n",
      "Topic: 6 Word: 0.002*\"music\" + 0.002*\"economics\" + 0.001*\"fitness\" + 0.001*\"food\" + 0.001*\"cities\" + 0.001*\"film\" + 0.001*\"economic\" + 0.001*\"city\" + 0.001*\"list\" + 0.001*\"album\"\n",
      "Topic: 7 Word: 0.003*\"film\" + 0.003*\"language\" + 0.001*\"languages\" + 0.001*\"chinese\" + 0.001*\"tamil\" + 0.001*\"literature\" + 0.001*\"museum\" + 0.001*\"music\" + 0.001*\"list\" + 0.001*\"award\"\n",
      "Topic: 8 Word: 0.002*\"film\" + 0.002*\"engineer\" + 0.002*\"university\" + 0.001*\"college\" + 0.001*\"technology\" + 0.001*\"award\" + 0.001*\"sciences\" + 0.001*\"institute\" + 0.001*\"minister\" + 0.001*\"management\"\n",
      "Topic: 9 Word: 0.004*\"music\" + 0.003*\"displaystyle\" + 0.002*\"album\" + 0.002*\"language\" + 0.001*\"languages\" + 0.001*\"song\" + 0.001*\"release\" + 0.001*\"program\" + 0.001*\"chart\" + 0.001*\"record\"\n"
     ]
    }
   ],
   "source": [
    "lda_model_tfidf = gensim.models.LdaMulticore(corpus_tfidf, num_topics=10, id2word=dictionary, passes=2, workers=4)\n",
    "for idx, topic in lda_model_tfidf.print_topics(-1):\n",
    "    print('Topic: {} Word: {}'.format(idx, topic))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#processed_docs[4310]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Score: 0.5162047743797302\t \n",
      "Topic: 0.007*\"social\" + 0.006*\"study\" + 0.005*\"theory\" + 0.004*\"science\" + 0.004*\"human\" + 0.004*\"philosophy\" + 0.004*\"self\" + 0.004*\"research\" + 0.003*\"society\" + 0.003*\"people\"\n",
      "\n",
      "Score: 0.31235620379447937\t \n",
      "Topic: 0.004*\"government\" + 0.004*\"century\" + 0.004*\"unite\" + 0.003*\"people\" + 0.003*\"world\" + 0.003*\"city\" + 0.003*\"economic\" + 0.003*\"empire\" + 0.003*\"south\" + 0.003*\"force\"\n",
      "\n",
      "Score: 0.0734516903758049\t \n",
      "Topic: 0.009*\"university\" + 0.007*\"program\" + 0.007*\"engineer\" + 0.005*\"school\" + 0.005*\"technology\" + 0.005*\"science\" + 0.004*\"college\" + 0.004*\"education\" + 0.004*\"research\" + 0.004*\"institute\"\n",
      "\n",
      "Score: 0.0635322704911232\t \n",
      "Topic: 0.008*\"food\" + 0.006*\"company\" + 0.005*\"market\" + 0.004*\"drink\" + 0.004*\"finance\" + 0.004*\"service\" + 0.003*\"world\" + 0.003*\"unite\" + 0.003*\"financial\" + 0.003*\"bank\"\n",
      "\n",
      "Score: 0.03414133936166763\t \n",
      "Topic: 0.006*\"self\" + 0.006*\"music\" + 0.006*\"culture\" + 0.006*\"people\" + 0.005*\"health\" + 0.004*\"psychology\" + 0.004*\"group\" + 0.003*\"study\" + 0.003*\"social\" + 0.003*\"form\"\n"
     ]
    }
   ],
   "source": [
    "#performance evaluation for BOW\n",
    "for index, score in sorted(lda_model[bow_corpus[4310]], key=lambda tup: -1*tup[1]):\n",
    "    print(\"\\nScore: {}\\t \\nTopic: {}\".format(score, lda_model.print_topic(index, 10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Score: 0.9881055951118469\t \n",
      "Topic: 0.002*\"social\" + 0.002*\"journal\" + 0.002*\"science\" + 0.002*\"psychology\" + 0.002*\"self\" + 0.001*\"philosophy\" + 0.001*\"anthropology\" + 0.001*\"study\" + 0.001*\"university\" + 0.001*\"economics\"\n",
      "\n",
      "Score: 0.011388801038265228\t \n",
      "Topic: 0.003*\"food\" + 0.003*\"finance\" + 0.003*\"transportation\" + 0.003*\"company\" + 0.002*\"drink\" + 0.002*\"service\" + 0.002*\"market\" + 0.002*\"global\" + 0.001*\"bank\" + 0.001*\"financial\"\n"
     ]
    }
   ],
   "source": [
    "#performance evaluation for TFIDF\n",
    "for index, score in sorted(lda_model_tfidf[bow_corpus[4310]], key=lambda tup: -1*tup[1]):\n",
    "    print(\"\\nScore: {}\\t \\nTopic: {}\".format(score, lda_model_tfidf.print_topic(index, 10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.8197623491287231\t Topic: 0.006*\"film\" + 0.006*\"series\" + 0.005*\"star\" + 0.004*\"television\" + 0.004*\"american\"\n",
      "Score: 0.0200318843126297\t Topic: 0.008*\"food\" + 0.006*\"company\" + 0.005*\"market\" + 0.004*\"drink\" + 0.004*\"finance\"\n",
      "Score: 0.020026803016662598\t Topic: 0.016*\"music\" + 0.004*\"record\" + 0.004*\"number\" + 0.003*\"release\" + 0.003*\"group\"\n",
      "Score: 0.020026493817567825\t Topic: 0.008*\"road\" + 0.007*\"county\" + 0.006*\"city\" + 0.005*\"highway\" + 0.004*\"north\"\n",
      "Score: 0.02002619206905365\t Topic: 0.019*\"language\" + 0.016*\"film\" + 0.007*\"languages\" + 0.005*\"write\" + 0.005*\"word\"\n",
      "Score: 0.02002609334886074\t Topic: 0.007*\"century\" + 0.007*\"literature\" + 0.006*\"write\" + 0.004*\"book\" + 0.004*\"displaystyle\"\n",
      "Score: 0.02002592943608761\t Topic: 0.006*\"self\" + 0.006*\"music\" + 0.006*\"culture\" + 0.006*\"people\" + 0.005*\"health\"\n",
      "Score: 0.020025119185447693\t Topic: 0.007*\"social\" + 0.006*\"study\" + 0.005*\"theory\" + 0.004*\"science\" + 0.004*\"human\"\n",
      "Score: 0.020024782046675682\t Topic: 0.009*\"university\" + 0.007*\"program\" + 0.007*\"engineer\" + 0.005*\"school\" + 0.005*\"technology\"\n",
      "Score: 0.02002429962158203\t Topic: 0.004*\"government\" + 0.004*\"century\" + 0.004*\"unite\" + 0.003*\"people\" + 0.003*\"world\"\n"
     ]
    }
   ],
   "source": [
    "unseen_document = 'Glossier is a makeup brand that every girl loves.'\n",
    "bow_vector = dictionary.doc2bow(preprocess(unseen_document))\n",
    "for index, score in sorted(lda_model_1[bow_vector], key=lambda tup: -1*tup[1]):\n",
    "    print(\"Score: {}\\t Topic: {}\".format(score, lda_model.print_topic(index, 5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.02002611),\n",
       " (1, 0.020025944),\n",
       " (2, 0.020025136),\n",
       " (3, 0.8197604),\n",
       " (4, 0.020024799),\n",
       " (5, 0.020033775),\n",
       " (6, 0.020024315),\n",
       " (7, 0.020026209),\n",
       " (8, 0.02002651),\n",
       " (9, 0.02002682)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(\"data/lda_model_1.pkl\", 'wb') as f:\n",
    "#      pickle.dump(lda_model, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/lda_model_1.pkl\", \"rb\") as input_file:\n",
    "    lda_model_1 = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_aggregate_score(lda_model, bow_corpus, i=0):\n",
    "    \"\"\"\n",
    "    Get the score for topics from wikipedia\n",
    "    \"\"\"\n",
    "\n",
    "    lda_all_docs = []   \n",
    "    \n",
    "    for i in range(len(bow_corpus)):\n",
    "        article = lda_model[bow_corpus[i]]\n",
    "    \n",
    "        if len(article) != 10:\n",
    "            indices = []\n",
    "            for item in article:\n",
    "                indices.append(item[0])\n",
    "            for index in list(range(10)):\n",
    "                if index not in indices:\n",
    "                    article.append((index, 0))\n",
    "        \n",
    "        lda_all_docs.append([i,article])\n",
    "\n",
    "\n",
    "    return lda_all_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregate_scores_mod1 = get_aggregate_score(lda_model, bow_corpus, i=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_df = pd.DataFrame(aggregate_scores_mod1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>[(1, 0.34407565), (2, 0.6388408), (0, 0), (3, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>[(2, 0.04269844), (3, 0.14525636), (5, 0.80569...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>[(3, 0.23120733), (5, 0.7549764), (0, 0), (1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>[(0, 0.21230577), (1, 0.03643345), (2, 0.34926...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>[(2, 0.27511144), (4, 0.040314987), (5, 0.0807...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                                                  1\n",
       "0  0  [(1, 0.34407565), (2, 0.6388408), (0, 0), (3, ...\n",
       "1  1  [(2, 0.04269844), (3, 0.14525636), (5, 0.80569...\n",
       "2  2  [(3, 0.23120733), (5, 0.7549764), (0, 0), (1, ...\n",
       "3  3  [(0, 0.21230577), (1, 0.03643345), (2, 0.34926...\n",
       "4  4  [(2, 0.27511144), (4, 0.040314987), (5, 0.0807..."
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_dicts = []\n",
    "for i in range(len(topics_df[1])):\n",
    "    one_doc = dict(topics_df[1][i])\n",
    "    list_of_dicts.append(one_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#list_of_dicts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_matrix_all_wiki = pd.DataFrame(list_of_dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(\"data/topic_matrix_all_wiki.pkl\", 'wb') as f:\n",
    "#      pickle.dump(topic_matrix_all_wiki, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/topic_matrix_all_wiki.pkl\", 'rb') as input_file:\n",
    "      topic_matrix_all_wiki = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_category = []\n",
    "for i in range(len(list_of_dicts)):\n",
    "    one_dict = list_of_dicts[i]\n",
    "    max_value = max(one_dict.values())\n",
    "    max_keys = [k for k, v in one_dict.items() if v == max_value]\n",
    "    best_category.append(max_keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#best_category "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "#[float(str(i).strip('[]')) for i in best_category]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_best_category = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_best_category['category'] = [int(str(i).strip('[]')) for i in best_category]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_best_category['article_index'] = range(len(category))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>article_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category  article_index\n",
       "0         2              0\n",
       "1         5              1\n",
       "2         5              2\n",
       "3         4              3\n",
       "4         6              4"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_best_category.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(\"df_best_category.pkl\", 'wb') as f:\n",
    "#      pickle.dump(df_best_category, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 3: TF-IDF Vectorization on 2000 dimensions\n",
    "Vectorize corpus and user input and return user input vector's most similar vectors in corpus.\n",
    "\n",
    "### Import necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import textstat\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from scipy.spatial.distance import cosine, cdist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textstat\n",
    "import operator\n",
    "import gensim\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
    "from nltk.stem.porter import *\n",
    "import numpy as np\n",
    "from operator import itemgetter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/new_corpus.pkl\", \"rb\") as input_file:\n",
    "    new_corpus = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/df_best_category.pkl\", \"rb\") as input_file:\n",
    "    df_best_category = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open (r\"data/dictionary.pkl\", \"rb\") as input_file: \n",
    "    dictionary = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open (r\"data/lda_model_1.pkl\", \"rb\") as input_file:\n",
    "    lda_model_1 = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>article_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category  article_index\n",
       "0         2              0\n",
       "1         5              1\n",
       "2         5              2\n",
       "3         4              3\n",
       "4         6              4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_best_category.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_vectorizer(pickle_file='data/vectorizer.pkl'):\n",
    "    \"\"\"Loads the trained TF/IDF vectorizer.\"\"\"\n",
    "    with open(pickle_file, 'rb') as f:\n",
    "        return pickle.load(f)\n",
    "    \n",
    "def load_corpus_vectors(pickle_file='data/corpus_vectors.pkl'):\n",
    "    \"\"\"Loads the corpus vectors.\"\"\"\n",
    "    with open(pickle_file, 'rb') as f:\n",
    "        return pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vocab_arr(vec):\n",
    "    n_features = len(vec.vocabulary_)\n",
    "    vocab_arr = np.empty(n_features, dtype=object)\n",
    "    for word, idx in vec.vocabulary_.items():\n",
    "        vocab_arr[idx] = word\n",
    "    return vocab_arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_k_vector(vector, feature_ranking, k=50):\n",
    "    \"\"\"Return the top k vector according to feature_ranking.\"\"\"\n",
    "    return vector[:, feature_ranking[:k]]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_50_text(text):\n",
    "    vec = load_vectorizer()\n",
    "    corpus_vectors = load_corpus_vectors().toarray()\n",
    "    sample_vector = vec.transform([text]).toarray()\n",
    "    feature_ranking = np.argsort(sample_vector[0])[::-1]\n",
    "    vocab_arr = get_vocab_arr(vec)\n",
    "    \n",
    "    distances = cdist(\n",
    "    get_top_k_vector(sample_vector, feature_ranking),\n",
    "    get_top_k_vector(corpus_vectors, feature_ranking),\n",
    "    )\n",
    "    \n",
    "    nearest_article_idxs = np.argsort(distances)\n",
    "    nearest_articles = new_corpus.loc[nearest_article_idxs[0], :]\n",
    "    top_50 = nearest_articles[:50]\n",
    "    \n",
    "    return top_50.sort_values(['score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = \"\"\"Data science is the study of the extraction of knowledge from data. It uses various techniques from many fields, including signal processing, mathematics, probability, machine learning, computer programming, statistics, data engineering, pattern matching, and data visualization, with the goal of extracting useful knowledge from the data. With computer systems able to handle more data, big data is an important aspect of data science.\n",
    "\n",
    "A person that does data science is called a data scientist. Data scientists solve complicated data problems using mathematics, statistics and computer science, although very good skill in these subjects are not required.[1] However, a data scientist is most likely to be an expert in only one or two of these disciplines, meaning that cross disciplinary teams can be a key component of data science.\n",
    "\n",
    "Good data scientists are able to apply their skills to achieve many kinds of purposes. Their skills and competencies vary widely.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = top_50_text(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>score</th>\n",
       "      <th>content</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13252</td>\n",
       "      <td>7.7</td>\n",
       "      <td>Oregon is a state located in the Western Unite...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1233</td>\n",
       "      <td>8.5</td>\n",
       "      <td>Walls And Mirrors is a computer science textbo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6277</td>\n",
       "      <td>12.0</td>\n",
       "      <td>In computing, the Global File System 2 or GFS2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11080</td>\n",
       "      <td>12.4</td>\n",
       "      <td>Globalize is a cross-platform JavaScript libra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2539</td>\n",
       "      <td>12.5</td>\n",
       "      <td>Pure Data (Pd) is a visual programming languag...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9969</td>\n",
       "      <td>13.1</td>\n",
       "      <td>Scientific Data is a peer-reviewed open access...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>12275</td>\n",
       "      <td>14.1</td>\n",
       "      <td>In computer science and computer programming, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>610</td>\n",
       "      <td>14.4</td>\n",
       "      <td>Digital anthropology is the anthropological st...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5831</td>\n",
       "      <td>14.6</td>\n",
       "      <td>A relational database is a digital database ba...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10459</td>\n",
       "      <td>14.7</td>\n",
       "      <td>Symbolic regression is a type of regression an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11804</td>\n",
       "      <td>15.6</td>\n",
       "      <td>A database is an organized collection of data,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11535</td>\n",
       "      <td>15.6</td>\n",
       "      <td>The General Social Survey (GSS) is a sociologi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12240</td>\n",
       "      <td>15.8</td>\n",
       "      <td>A data model (or datamodel) is an abstract mod...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1678</td>\n",
       "      <td>16.2</td>\n",
       "      <td>Most of the terms listed in Wikipedia glossari...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1784</td>\n",
       "      <td>16.3</td>\n",
       "      <td>LANSA is an integrated development environment...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>582</td>\n",
       "      <td>16.9</td>\n",
       "      <td>Conflict and Mediation Event Observations (CAM...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>10389</td>\n",
       "      <td>16.9</td>\n",
       "      <td>In computer science, a collection or container...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7068</td>\n",
       "      <td>17.0</td>\n",
       "      <td>Wireless sensor network (WSN) refers to a grou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>7335</td>\n",
       "      <td>17.1</td>\n",
       "      <td>In object oriented programming languages,   en...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>9934</td>\n",
       "      <td>17.2</td>\n",
       "      <td>R is a programming language and free software ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>8998</td>\n",
       "      <td>17.2</td>\n",
       "      <td>Osteoware is a free data recording software fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>9193</td>\n",
       "      <td>17.3</td>\n",
       "      <td>A column-oriented DBMS (or columnar database m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1186</td>\n",
       "      <td>17.4</td>\n",
       "      <td>Qualitative research is a scientific method of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2785</td>\n",
       "      <td>17.5</td>\n",
       "      <td>Behavioral analytics is a recent advancement i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2708</td>\n",
       "      <td>17.7</td>\n",
       "      <td>In computer science, an abstract data type (AD...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>6834</td>\n",
       "      <td>18.0</td>\n",
       "      <td>The following table of United States cities by...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>6182</td>\n",
       "      <td>18.1</td>\n",
       "      <td>Rare or extreme events are events that occur w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>6826</td>\n",
       "      <td>18.2</td>\n",
       "      <td>Before ForDisc, many anthropologists based the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>8130</td>\n",
       "      <td>18.4</td>\n",
       "      <td>Data science is a multi-disciplinary field tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>7536</td>\n",
       "      <td>18.7</td>\n",
       "      <td>The Quantified Self refers both to the cultura...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>7447</td>\n",
       "      <td>18.8</td>\n",
       "      <td>Information technology (IT) is the use of comp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>6868</td>\n",
       "      <td>18.9</td>\n",
       "      <td>Statistical geography is the study and practic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>9603</td>\n",
       "      <td>19.1</td>\n",
       "      <td>In natural and social sciences, and sometimes ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>10643</td>\n",
       "      <td>19.3</td>\n",
       "      <td>The Global Consciousness Project (GCP, also ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>6962</td>\n",
       "      <td>19.7</td>\n",
       "      <td>Shlomo Argamon is an American/Israeli computer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>14008</td>\n",
       "      <td>20.1</td>\n",
       "      <td>Transportation forecasting is the attempt of e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>5492</td>\n",
       "      <td>20.4</td>\n",
       "      <td>The Global Data Synchronization Network (GDSN)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>10891</td>\n",
       "      <td>20.4</td>\n",
       "      <td>A geographic information system (GIS) is a sys...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>2635</td>\n",
       "      <td>20.8</td>\n",
       "      <td>In computing and telecommunications, the paylo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>7678</td>\n",
       "      <td>20.9</td>\n",
       "      <td>The GESIS – Leibniz Institute for the Social S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>4765</td>\n",
       "      <td>21.3</td>\n",
       "      <td>Intuitive statistics, or folk statistics, refe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>12287</td>\n",
       "      <td>21.5</td>\n",
       "      <td>This is a list of sovereign states and territo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>9470</td>\n",
       "      <td>21.6</td>\n",
       "      <td>Inmarsat plc (LSE: ISAT) is a British satellit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>9831</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Behavioral health outcome management (BHOM) in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>10068</td>\n",
       "      <td>23.8</td>\n",
       "      <td>In mathematical analysis and computer science,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>10333</td>\n",
       "      <td>26.3</td>\n",
       "      <td>Low-complexity art, first described by Jürgen ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>11864</td>\n",
       "      <td>53.4</td>\n",
       "      <td>SDS may refer to:\\n\\nOrganizations\\nBusinesses...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>6074</td>\n",
       "      <td>60.0</td>\n",
       "      <td>In computing the Process Environment Block (ab...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>444</td>\n",
       "      <td>74.6</td>\n",
       "      <td>Vflo is a commercially available, physics-base...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>8029</td>\n",
       "      <td>135.5</td>\n",
       "      <td>Stack may refer to:\\n\\nScience\\nStack (mathema...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    index  score                                            content\n",
       "0   13252    7.7  Oregon is a state located in the Western Unite...\n",
       "1    1233    8.5  Walls And Mirrors is a computer science textbo...\n",
       "2    6277   12.0  In computing, the Global File System 2 or GFS2...\n",
       "3   11080   12.4  Globalize is a cross-platform JavaScript libra...\n",
       "4    2539   12.5  Pure Data (Pd) is a visual programming languag...\n",
       "5    9969   13.1  Scientific Data is a peer-reviewed open access...\n",
       "6   12275   14.1  In computer science and computer programming, ...\n",
       "7     610   14.4  Digital anthropology is the anthropological st...\n",
       "8    5831   14.6  A relational database is a digital database ba...\n",
       "9   10459   14.7  Symbolic regression is a type of regression an...\n",
       "10  11804   15.6  A database is an organized collection of data,...\n",
       "11  11535   15.6  The General Social Survey (GSS) is a sociologi...\n",
       "12  12240   15.8  A data model (or datamodel) is an abstract mod...\n",
       "13   1678   16.2  Most of the terms listed in Wikipedia glossari...\n",
       "14   1784   16.3  LANSA is an integrated development environment...\n",
       "15    582   16.9  Conflict and Mediation Event Observations (CAM...\n",
       "16  10389   16.9  In computer science, a collection or container...\n",
       "17   7068   17.0  Wireless sensor network (WSN) refers to a grou...\n",
       "18   7335   17.1  In object oriented programming languages,   en...\n",
       "19   9934   17.2  R is a programming language and free software ...\n",
       "20   8998   17.2  Osteoware is a free data recording software fo...\n",
       "21   9193   17.3  A column-oriented DBMS (or columnar database m...\n",
       "22   1186   17.4  Qualitative research is a scientific method of...\n",
       "23   2785   17.5  Behavioral analytics is a recent advancement i...\n",
       "24   2708   17.7  In computer science, an abstract data type (AD...\n",
       "25   6834   18.0  The following table of United States cities by...\n",
       "26   6182   18.1  Rare or extreme events are events that occur w...\n",
       "27   6826   18.2  Before ForDisc, many anthropologists based the...\n",
       "28   8130   18.4  Data science is a multi-disciplinary field tha...\n",
       "29   7536   18.7  The Quantified Self refers both to the cultura...\n",
       "30   7447   18.8  Information technology (IT) is the use of comp...\n",
       "31   6868   18.9  Statistical geography is the study and practic...\n",
       "32   9603   19.1  In natural and social sciences, and sometimes ...\n",
       "33  10643   19.3  The Global Consciousness Project (GCP, also ca...\n",
       "34   6962   19.7  Shlomo Argamon is an American/Israeli computer...\n",
       "35  14008   20.1  Transportation forecasting is the attempt of e...\n",
       "36   5492   20.4  The Global Data Synchronization Network (GDSN)...\n",
       "37  10891   20.4  A geographic information system (GIS) is a sys...\n",
       "38   2635   20.8  In computing and telecommunications, the paylo...\n",
       "39   7678   20.9  The GESIS – Leibniz Institute for the Social S...\n",
       "40   4765   21.3  Intuitive statistics, or folk statistics, refe...\n",
       "41  12287   21.5  This is a list of sovereign states and territo...\n",
       "42   9470   21.6  Inmarsat plc (LSE: ISAT) is a British satellit...\n",
       "43   9831   22.0  Behavioral health outcome management (BHOM) in...\n",
       "44  10068   23.8  In mathematical analysis and computer science,...\n",
       "45  10333   26.3  Low-complexity art, first described by Jürgen ...\n",
       "46  11864   53.4  SDS may refer to:\\n\\nOrganizations\\nBusinesses...\n",
       "47   6074   60.0  In computing the Process Environment Block (ab...\n",
       "48    444   74.6  Vflo is a commercially available, physics-base...\n",
       "49   8029  135.5  Stack may refer to:\\n\\nScience\\nStack (mathema..."
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_level_change(x,text):\n",
    "    \"\"\"\n",
    "    Takes in a value and returns the article with the score closest to that value.\n",
    "    \"\"\"\n",
    "    top_50_df = top_50_text(text)\n",
    "    top_50_df = top_50_df.reset_index()\n",
    "    top_50_dict = top_50_df['score'].to_dict()\n",
    "    abs_values = {}\n",
    "    for key, value in top_50_dict.items():\n",
    "        temp = abs(value-x)\n",
    "        abs_values.update({key:temp})\n",
    "    article_id = min(abs_values, key=abs_values.get)\n",
    "    level_change = top_50_df['content'][article_id]\n",
    "    return level_change\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Wireless sensor network (WSN) refers to a group of spatially dispersed and dedicated sensors for monitoring and recording the physical conditions of the environment and organizing the collected data at a central location. WSNs measure environmental conditions like temperature, sound, pollution levels, humidity, wind, and so on.\\nThese are similar to wireless ad hoc networks in the sense that they rely on wireless connectivity and spontaneous formation of networks so that sensor data can be transported wirelessly. WSNs are spatially distributed autonomous sensors to monitor physical or environmental conditions, such as temperature, sound, pressure, etc. and to cooperatively pass their data through the network to a main location. The more modern networks are bi-directional, also enabling control of sensor activity.  The development of wireless sensor networks was motivated by military applications such as battlefield surveillance; today such networks are used in many industrial and consumer applications, such as industrial process monitoring and control, machine health monitoring, and so on.\\nThe WSN is built of \"nodes\" – from a few to several hundreds or even thousands, where each node is connected to one (or sometimes several) sensors. Each such sensor network node has typically several parts: a radio transceiver with an internal antenna or connection to an external antenna, a microcontroller, an electronic circuit for interfacing with the sensors and an energy source, usually a battery or an embedded form of energy harvesting. A sensor node might vary in size from that of a shoebox down to the size of a grain of dust, although functioning \"motes\" of genuine microscopic dimensions have yet to be created. The cost of sensor nodes is similarly variable, ranging from a few to hundreds of dollars, depending on the complexity of the individual sensor nodes. Size and cost constraints on sensor nodes result in corresponding constraints on resources such as energy, memory, computational speed and communications bandwidth. The topology of the WSNs can vary from a simple star network to an advanced multi-hop wireless mesh network. The propagation technique between the hops of the network can be routing or flooding.In computer science and telecommunications, wireless sensor networks are an active research area with numerous workshops and conferences arranged each year, for example IPSN, SenSys, and EWSN.\\n\\nApplication\\nArea monitoring\\nArea monitoring is a common application of WSNs. In area monitoring, the WSN is deployed over a region where some phenomenon is to be monitored. A military example is the use of sensors to detect enemy intrusion; a civilian example is the geo-fencing of gas or oil pipelines.\\n\\nHealth care monitoring\\nThere are several types of sensor networks for medical applications: implanted, wearable, and environment-embedded. Implantable medical devices are those that are inserted inside the human body. Wearable devices are used on the body surface of a human or just at close proximity of the user. Environment-embedded systems employ sensors contained in the environment. Possible applications include body position measurement, location of persons, overall monitoring of ill patients in hospitals and at home. Devices embedded in the environment track the physical state of a person for continuous health diagnosis, using as input the data from a network of depth cameras, a sensing floor, or other similar devices. Body-area networks can collect information about an individual\\'s health, fitness, and energy expenditure. In health care applications the privacy and authenticity of user data has prime importance. Especially due to the integration of sensor networks, with IoT, the user authentication becomes more challenging; however, a solution is presented in recent work.\\n\\nEnvironmental/Earth sensing\\nThere are many applications in monitoring environmental parameters, examples of which are given below. They share the extra challenges of harsh environments and reduced power supply.\\n\\nAir pollution monitoring\\nWireless sensor networks have been deployed in several cities (Stockholm, London, and Brisbane) to monitor the concentration of dangerous gases for citizens. These can take advantage of the ad hoc wireless links rather than wired installations, which also make them more mobile for testing readings in different areas.\\n\\nForest fire detection\\nA network of Sensor Nodes can be installed in a forest to detect when a fire has started. The nodes can be equipped with sensors to measure temperature, humidity and gases which are produced by fire in the trees or vegetation. The early detection is crucial for a successful action of the firefighters; thanks to Wireless Sensor Networks, the fire brigade will be able to know when a fire is started and how it is spreading.\\n\\nLandslide detection\\nA landslide detection system makes use of a wireless sensor network to detect the slight movements of soil and changes in various parameters that may occur before or during a landslide. Through the data gathered it may be possible to know the impending occurrence of landslides long before it actually happens.\\n\\nWater quality monitoring\\nWater quality monitoring involves analyzing water properties in dams, rivers, lakes and oceans, as well as underground water reserves. The use of many wireless distributed sensors enables the creation of a more accurate map of the water status, and allows the permanent deployment of monitoring stations in locations of difficult access, without the need of manual data retrieval.\\n\\nNatural disaster prevention\\nWireless sensor networks can be effective in preventing adverse consequences of natural disasters, like floods. Wireless nodes have been deployed successfully in rivers, where changes in water levels must be monitored in real time.\\n\\nIndustrial monitoring\\nMachine health monitoring\\nWireless sensor networks have been developed for machinery condition-based maintenance (CBM) as they offer significant cost savings and enable new functionality.Wireless sensors can be placed in locations difficult or impossible to reach with a wired system, such as rotating machinery and untethered vehicles.\\n\\nData logging\\nWireless sensor networks also are used for the collection of data for monitoring of environmental information. This can be as simple as monitoring the temperature in a fridge or the level of water in overflow tanks in nuclear power plants. The statistical information can then be used to show how systems have been working. The advantage of WSNs over conventional loggers is the \"live\" data feed that is possible.\\n\\nWater/waste water monitoring\\nMonitoring the quality and level of water includes many activities such as checking the quality of underground or surface water and ensuring a country’s water infrastructure for the benefit of both human and animal. It may be used to protect the wastage of water.\\n\\nStructural health monitoring\\nWireless sensor networks can be used to monitor the condition of civil infrastructure and related geo-physical processes close to real time, and over long periods through data logging, using appropriately interfaced sensors.\\n\\nWine production\\nWireless sensor networks are used to monitor wine production, both in the field and the cellar.\\n\\nThreat detection\\nThe Wide Area Tracking System (WATS) is a prototype network for detecting a ground-based nuclear device such as a nuclear \"briefcase bomb.\" WATS is being developed at the Lawrence Livermore National Laboratory (LLNL). WATS would be made up of wireless gamma and neutron sensors connected through a communications network. Data picked up by the sensors undergoes \"data fusion\", which converts the information into easily interpreted forms; this data fusion is the most important aspect of the system.The data fusion process occurs within the sensor network rather than at a centralized computer and is performed by a specially developed algorithm based on Bayesian statistics. WATS would not use a centralized computer for analysis because researchers found that factors such as latency and available bandwidth tended to create significant bottlenecks. Data processed in the field by the network itself (by transferring small amounts of data between neighboring sensors) is faster and makes the network more scalable.An important factor in WATS development is ease of deployment, since more sensors both improves the detection rate and reduces false alarms. WATS sensors could be deployed in permanent positions or mounted in vehicles for mobile protection of specific locations. One barrier to the implementation of WATS is the size, weight, energy requirements and cost of currently available wireless sensors. The development of improved sensors is a major component of current research at the Nonproliferation, Arms Control, and International Security (NAI) Directorate at LLNL.\\nWATS was profiled to the U.S. House of Representatives\\' Military Research and Development Subcommittee on October 1, 1997 during a hearing on nuclear terrorism and countermeasures. On August 4, 1998 in a subsequent meeting of that subcommittee, Chairman Curt Weldon stated that research funding for WATS had been cut by the Clinton administration to a subsistence level and that the program had been poorly re-organized.\\n\\nCharacteristics\\nThe main characteristics of a WSN include\\n\\nPower consumption constraints for nodes using batteries or energy harvesting. Examples of suppliers are ReVibe Energy and Perpetuum\\nAbility to cope with node failures (resilience)\\nSome mobility of nodes (for highly mobile nodes see MWSNs)\\nHeterogeneity of nodes\\nHomogeneity of nodes\\nScalability to large scale of deployment\\nAbility to withstand harsh environmental conditions\\nEase of use\\nCross-layer designCross-layer is becoming an important studying area for wireless communications. In addition, the traditional layered approach presents three main problems:\\n\\nTraditional layered approach cannot share different information among different layers, which leads to each layer not having complete information. The traditional layered approach cannot guarantee the optimization of the entire network.\\nThe traditional layered approach does not have the ability to adapt to the environmental change.\\nBecause of the interference between the different users, access conflicts, fading, and the change of environment in the wireless sensor networks, traditional layered approach for wired networks is not applicable to wireless networks.So the cross-layer can be used to make the optimal modulation to improve the transmission performance, such as data rate, energy efficiency, QoS (Quality of Service), etc. Sensor nodes can be imagined as small computers which are extremely basic in terms of their interfaces and their components. They usually consist of a processing unit with limited computational power and limited memory, sensors or MEMS (including specific conditioning circuitry), a communication device (usually radio transceivers or alternatively optical), and a power source usually in the form of a battery. Other possible inclusions are energy harvesting modules, secondary ASICs, and possibly secondary communication interface (e.g. RS-232 or USB).\\nThe base stations are one or more components of the WSN with much more computational, energy and communication resources. They act as a gateway between sensor nodes and the end user as they typically forward data from the WSN on to a server. Other special components in routing based networks are routers, designed to compute, calculate and distribute the routing tables.\\n\\nPlatforms\\nHardware\\nOne major challenge in a WSN is to produce low cost and tiny sensor nodes. There are an increasing number of small companies producing WSN hardware and the commercial situation can be compared to home computing in the 1970s. Many of the nodes are still in the research and development stage, particularly their software. Also inherent to sensor network adoption is the use of very low power methods for radio communication and data acquisition.\\nIn many applications, a WSN communicates with a Local Area Network or Wide Area Network through a gateway. The Gateway acts as a bridge between the WSN and the other network. This enables data to be stored and processed by devices with more resources, for example, in a remotely located server. A wireless wide area network used primarily for low-power devices is known as a Low-Power Wide-Area Network (LPWAN).\\n\\nWireless\\nThere are several wireless standards and solutions for sensor node connectivity. Thread and ZigBee can connect sensors operating at 2.4 GHz with a data rate of 250kbit/s. Many use a lower frequency to increase radio range (typically 1 km), for example Z-wave operates at 915 MHz and in the EU 868 MHz has been widely used but these have a lower data rate (typically 50 kb/s). The IEEE 802.15.4 working group provides a standard for low power device connectivity and commonly sensors and smart meters use one of these standards for connectivity. With the emergence of Internet of Things, many other proposals have been made to provide sensor connectivity. LORA is a form of LPWAN which provides long range low power wireless connectivity for devices, which has been used in smart meters. Wi-SUN connects devices at home. NarrowBand IOT and LTE-M can connect up to millions of sensors and devices using cellular technology.\\n\\nSoftware\\nEnergy is the scarcest resource of WSN nodes, and it determines the lifetime of WSNs. WSNs may be deployed in large numbers in various environments, including remote and hostile regions, where ad hoc communications are a key component. For this reason, algorithms and protocols need to address the following issues:\\n\\nIncreased lifespan\\nRobustness and fault tolerance\\nSelf-configurationLifetime maximization: Energy/Power Consumption of the sensing device should be minimized and sensor nodes should be energy efficient since their limited energy resource determines their lifetime. To conserve power, wireless sensor nodes normally power off both the radio transmitter and the radio receiver when not in use.\\n\\nRouting Protocols\\nWireless sensor networks are composed of low-energy, small-size, and low-range unattended sensor nodes. Recently, it has been observed that by periodically turning on and off the sensing and communication capabilities of sensor nodes, we can significantly reduce the active time and thus prolong network lifetime. However, this duty cycling may result in high network latency, routing overhead, and neighbor discovery delays due to asynchronous sleep and wake-up scheduling. These limitations call for a countermeasure for duty-cycled wireless sensor networks which should minimize routing information, routing traffic load, and energy consumption. Researchers from Sungkyunkwan University have proposed a lightweight non-increasing delivery-latency interval routing referred as LNDIR. This scheme can discover minimum latency routes at each non-increasing delivery-latency interval instead of each time slot. Simulation experiments demonstrated the validity of this novel approach in minimizing routing information stored at each sensor. Furthermore, this novel routing can also guarantee the minimum delivery latency from each source to the sink. Performance improvements of up to 12-fold and 11-fold are observed in terms of routing traffic load reduction and energy efficiency, respectively, as compared to existing schemes.\\n\\nOperating systems\\nOperating systems for wireless sensor network nodes are typically less complex than general-purpose operating systems. They more strongly resemble embedded systems, for two reasons. First, wireless sensor networks are typically deployed with a particular application in mind, rather than as a general platform. Second, a need for low costs and low power leads most wireless sensor nodes to have low-power microcontrollers ensuring that mechanisms such as virtual memory are either unnecessary or too expensive to implement.\\nIt is therefore possible to use embedded operating systems such as eCos or uC/OS for sensor networks. However, such operating systems are often designed with real-time properties.\\nTinyOS is perhaps the first operating system specifically designed for wireless sensor networks. TinyOS is based on an event-driven programming model instead of multithreading. TinyOS programs are composed of event handlers and tasks with run-to-completion semantics. When an external event occurs, such as an incoming data packet or a sensor reading, TinyOS signals the appropriate event handler to handle the event. Event handlers can post tasks that are scheduled by the TinyOS kernel some time later.\\nLiteOS is a newly developed OS for wireless sensor networks, which provides UNIX-like abstraction and support for the C programming language.\\nContiki is an OS which uses a simpler programming style in C while providing advances such as 6LoWPAN and Protothreads.\\nRIOT (operating system) is a more recent real-time OS including similar functionality to Contiki.\\nPreonVM is an OS for wireless sensor networks, which provides 6LoWPAN based on Contiki and support for the Java programming language.\\n\\nOnline collaborative sensor data management platforms\\nOnline collaborative sensor data management platforms are on-line database services that allow sensor owners to register and connect their devices to feed data into an online database for storage and also allow developers to connect to the database and build their own applications based on that data. Examples include Xively and the Wikisensing platform. Such platforms simplify online collaboration between users over diverse data sets ranging from energy and environment data to that collected from transport services. Other services include allowing developers to embed real-time graphs & widgets in websites; analyse and process historical data pulled from the data feeds; send real-time alerts from any datastream to control scripts, devices and environments.\\nThe architecture of the Wikisensing system describes the key components of such systems to include APIs and interfaces for online collaborators, a middleware containing the business logic needed for the sensor data management and processing and a storage model suitable for the efficient storage and retrieval of large volumes of data.\\n\\nSimulation\\nAt present, agent-based modeling and simulation is the only paradigm which allows the simulation of complex behavior in the environments of wireless sensors (such as flocking). Agent-based simulation of wireless sensor and ad hoc networks is a relatively new paradigm. Agent-based modelling was originally based on social simulation.\\nNetwork simulators like Opnet, Tetcos NetSim and NS can be used to simulate a wireless sensor network.\\n\\nOther concepts\\nSecurity\\nInfrastructure-less architecture (i.e. no gateways are included, etc.) and inherent requirements (i.e. unattended working environment, etc.) of WSNs might pose several weak points that attract adversaries. Therefore, security is a big concern when WSNs are deployed for special applications such as military and healthcare. Owing to their unique characteristics, traditional security methods of computer networks would be useless (or less effective) for WSNs. Hence, lack of security mechanisms would cause intrusions towards those networks. These intrusions need to be detected and mitigation methods should be applied.\\n\\nDistributed sensor network\\nIf a centralized architecture is used in a sensor network and the central node fails, then the entire network will collapse, however the reliability of the sensor network can be increased by using a distributed control architecture. Distributed control is used in WSNs for the following reasons:\\n\\nSensor nodes are prone to failure,\\nFor better collection of data,\\nTo provide nodes with backup in case of failure of the central node.There is also no centralised body to allocate the resources and they have to be self organized.\\n\\nData integration and sensor web\\nThe data gathered from wireless sensor networks is usually saved in the form of numerical data in a central base station. Additionally, the Open Geospatial Consortium (OGC) is specifying standards for interoperability interfaces and metadata encodings that enable real time integration of heterogeneous sensor webs into the Internet, allowing any individual to monitor or control wireless sensor networks through a web browser.\\n\\nIn-network processing\\nTo reduce communication costs some algorithms remove or reduce nodes\\' redundant sensor information and avoid forwarding data that is of no use. This technique has been used, for instance, for distributed anomaly detection or distributed optimization. As nodes can inspect the data they forward, they can measure averages or directionality for example of readings from other nodes. For example, in sensing and monitoring applications, it is generally the case that neighboring sensor nodes monitoring an environmental feature typically register similar values. This kind of data redundancy due to the spatial correlation between sensor observations inspires techniques for in-network data aggregation and mining. Aggregation reduces the amount of network traffic which helps to reduce energy consumption on sensor nodes. Recently, it has been found that network gateways also play an important role in improving energy efficiency of sensor nodes by scheduling more resources for the nodes with more critical energy efficiency need and advanced energy efficient scheduling algorithms need to be implemented at network gateways for the improvement of the overall network energy efficiency.\\n\\nSecure data aggregation\\nThis is a form of in-network processing where sensor nodes are assumed to be unsecured with limited available energy, while the base station is assumed to be secure with unlimited available energy. Aggregation complicates the already existing security challenges for wireless sensor networks and requires new security techniques tailored specifically for this scenario. Providing security to aggregate data in wireless sensor networks is known as secure data aggregation in WSN. were the first few works discussing techniques for secure  data aggregation in wireless sensor networks.\\nTwo main security challenges in secure data aggregation are confidentiality and integrity of data. While encryption is traditionally used to provide end to end confidentiality in wireless sensor network, the aggregators in a secure data aggregation scenario need to decrypt the encrypted data to perform aggregation. This exposes the plaintext at the aggregators, making the data vulnerable to attacks from an adversary. Similarly an aggregator can inject false data into the aggregate and make the base station accept false data. Thus, while data aggregation improves energy efficiency of a network, it complicates the existing security challenges.\\n\\nSee also\\nAutonomous system\\nBluetooth mesh networking\\nCenter for Embedded Network Sensing\\nList of ad hoc routing protocols\\nMobile wireless sensor networks\\nOpenWSN\\nOptical wireless communications\\nRobotic mapping\\nSmart, connected products\\nWireless ad hoc networks\\n\\nReferences\\nFurther reading\\nKiran Maraiya, Kamal Kant, Nitin Gupta \"Wireless Sensor Network: A Review on Data Aggregation\" International Journal of Scientific & Engineering Research Volume 2 Issue 4, April 2011.\\nChalermek Intanagonwiwat, Deborah Estrin, Ramesh Govindan, John Heidemann, \"Impact of Network Density on Data Aggregation in Wireless SensorNetworks,\" November 4, 2001.\\n\\nExternal links\\nIEEE 802.15.4 Standardization Committee\\nSecure Data Aggregation in Wireless Sensor Networks: A *Survey\\nA list of secure aggregation proposals for WSN'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_level_change(17,sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 4: TF-IDF Vectorization on 2000 dimensions with top 20 features\n",
    "It turns out that specifying cosine distance to the top 20 features returns results that are more similar to the user input text than using all features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import necessary packages "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from scipy.spatial.distance import cosine, cdist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"data/df_corpus2.pkl\", \"rb\") as input_file:\n",
    "    df_corpus2 = pickle.load(input_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = TfidfVectorizer(stop_words='english', max_features=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = vec.fit(df_corpus2['content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#vec.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus2_vectors = vec.transform(df_corpus2['content']).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_doc = \"A watermelon is a type of edible fruit, they are 92% water. About 6% of a watermelon is sugar, which makes it very sweet. There are many different types of watermelon. Some have a green rind on the outside and a red-pink flesh on the inside, with brown seeds. Some can have yellow flesh, and some can be seedless. The green rind on the outside is not usually eaten, though it can be used as a vegetable. It can also be stewed or pickled. Most watermelons are oblong or spherical. In Japan, watermelons are grown in different shapes. Many people like to eat watermelon in the summer because the fruit is cool and refreshing. Watermelons are a great source of vitamin A, vitamin C, vitamin B6 and vitamin B1. They also contain potassium, magnesium, carotenoid antioxidant, and lycopene. Watermelons are fruits that come from a vine-like plant.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_doc_vector = vec.transform([user_doc]).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.72840959, 0.        , 0.        , 0.68514193, 0.        ]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_doc_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "distances = cdist(user_doc_vector,\n",
    "                  corpus2_vectors,\n",
    "                  metric='cosine')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14216,)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distances.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranking = np.argsort(distances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14216,)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranking.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3879,  7216,  8986,  1943, 11400, 10636,  9629,  8609,  5384,\n",
       "        1488])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_ten = ranking[:10]\n",
    "top_ten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distances[top_ten]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3879     With a climate as varied and extreme as India,...\n",
       "7216     International finance (also referred to as int...\n",
       "8986     A pakudos is a visual motif used by the Mangya...\n",
       "1943     Construct (psychology), also hypothetical cons...\n",
       "11400    The epicanthic fold is the skin fold of the up...\n",
       "10636    Chuan (Chinese: 串, Dungan: Чўан, pinyin: chuàn...\n",
       "9629     Global Village Tech Park is a software technol...\n",
       "8609     Tejuíno is a cold beverage made from fermented...\n",
       "5384     Gens was used by Lewis H. Morgan (in Ancient S...\n",
       "1488     \"Natural Science\" is a song by the Canadian ro...\n",
       "10581    Argument to moderation (Latin: argumentum ad t...\n",
       "8306     Burgess may refer to:\\n\\nPeople\\nBurgess (name...\n",
       "11978    Trust for Nature is a not-for-profit organisat...\n",
       "7376     Kochi is a city in the Indian state of Kerala....\n",
       "7953     A self-directed Individual Retirement Account ...\n",
       "6197     Vets Prevail, is a veteran-created online ment...\n",
       "13135    Humans of some cultures eat octopus. The arms ...\n",
       "3660     Anam Cara is a phrase that refers to the Celti...\n",
       "12933    Geographical features are naturally-created fe...\n",
       "10337    The Johari window is a technique that helps pe...\n",
       "13278    Chinta (Sanskrit: चिन्ता) in Hindu philosophy ...\n",
       "7205     Quantitative analysis of behavior is the appli...\n",
       "3499     Thai tea is a Thai drink made from tea, milk a...\n",
       "4408     Paya is a traditional food from the Indian Sub...\n",
       "2417     Common goods are defined in economics as goods...\n",
       "2111     Ikigai (生き甲斐, pronounced [ikiɡai]) is a Japane...\n",
       "10943    A substitute good is a good that can be used i...\n",
       "5471     The 2009 Collier Township shooting, also refer...\n",
       "12954    A showreel (also known as a demo reel or sizzl...\n",
       "3148     A somatic symptom disorder, formerly known as ...\n",
       "                               ...                        \n",
       "8910     Handia  (Also handi or hadiya) is a rice beer ...\n",
       "8919     The Department of Housing, Planning and Local ...\n",
       "8920     The Nature of the Beast is the ninth studio al...\n",
       "8924     This is a list of living actors from the Golde...\n",
       "8926     Drevlyans Nature Reserve (Ukrainian: Древлянсь...\n",
       "8927     Nature Chemistry is a monthly peer-reviewed sc...\n",
       "8935     College of Science and Technology - University...\n",
       "8792     The Australian Journal of Anthropology is a pe...\n",
       "8559     Addictive Behaviors is a monthly peer-reviewed...\n",
       "8747     The Ministry of Finance of Yugoslavia refers t...\n",
       "8744     Prestegårdsskogen Nature Reserve (Norwegian: P...\n",
       "8594     Movies With Manings was a Canadian movie telev...\n",
       "8596     Bruce Kent Bateman is an American movie produc...\n",
       "8615     This is a list of colleges and universities in...\n",
       "8616     Cordial may refer to:\\nFood and drink:\\n\\nLiqu...\n",
       "8632     Lauren Christine German (born November 29, 197...\n",
       "8643     Yemen Chronicle: An Anthropology of War and Me...\n",
       "8644     Ketan Mulmuley is a professor in the Departmen...\n",
       "8654     Pharmacology Biochemistry and Behavior is a mo...\n",
       "8670     Ozark Actors Theatre is an Equity theater comp...\n",
       "8674     The Ministry of Environment (Portuguese: Minis...\n",
       "8677     For the most populous cities in North America,...\n",
       "8680     Attack of the Movies 3D is a rail shooter for ...\n",
       "8682     Zeekoevlei is a major freshwater lake on the C...\n",
       "8701     Turner Classic Movies, a movie-oriented networ...\n",
       "8711     Beshtash Nature Park is a nature park in Talas...\n",
       "8720     National nature reserves in Suffolk, England a...\n",
       "8733     Eli Global is a multinational corporation base...\n",
       "8745     Rain City may refer to the following cities:\\n...\n",
       "14215    Doubles is a common street food of Trinidadian...\n",
       "Name: content, Length: 14216, dtype: object"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_corpus2.loc[ranking, 'content']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textstat\n",
    "import operator\n",
    "import wikipedia\n",
    "import wikipediaapi\n",
    "\n",
    "def give_harder_level(text):\n",
    "    \"\"\"\n",
    "    Takes in text and returns a harder level read\n",
    "    \"\"\"\n",
    "    ##find all the articles that are easier than this article \n",
    "    all_harder_text = []\n",
    "    input_score = textstat.flesch_kincaid_grade(text)\n",
    "    for i in range(len(df_corpus2['score'])):\n",
    "        if df_corpus2['score'][i] > input_score:\n",
    "            all_harder_text.append(df_corpus2['content'][i])\n",
    "    \n",
    "    vec = TfidfVectorizer(stop_words='english', max_features=2000)\n",
    "    vec = vec.fit(all_harder_text)\n",
    "    corpus2_vectors = vec.transform(all_harder_text).toarray()\n",
    "    user_doc = text\n",
    "    user_doc_vector = vec.transform([user_doc]).toarray() \n",
    "    user_doc_vector_short = np.argsort(user_doc_vector)[-25:]\n",
    "    distances = cdist(user_doc_vector_short,\n",
    "                  corpus2_vectors[-25:],\n",
    "                  metric='cosine')[0]\n",
    "    ranking = np.argsort(distances)\n",
    "    top = ranking[0]\n",
    "    best_match = df_corpus2['content'][top]\n",
    "        \n",
    "    print(distances[top])\n",
    "    return (best_match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = \"On one level, this is a business dispute that highlights tensions in the music industry (both intractably ancient and very current). When Swift signed her six-album deal that furnished Big Machine with rights to her masters, she was partaking in a classic arrangement for new artists: handing over future control of music in exchange for start-up promotional, recording, and distribution help. As she’s risen to megastardom, she’s chafed at that arrangement the same way that many successful musicians have chafed at not having ownership of their work. Prince, who famously protested Warner Bros. Records in 1993 by writing the word slave on his cheek, struck a deal for his own back catalog at great cost in 2014. Paul McCartney preached for years about the importance of artists owning music—and one person he preached to, Michael Jackson, ended up buying the rights to the Beatles’ catalog from under him. West himself recently filed suit to gain control of his own masters.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6822204939076502\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Value or values may refer to:\\n\\nValue (philosophy),\\nValue (ethics) it may be described as treating actions themselves as abstract objects, putting value to them\\nValues (Western philosophy) expands the notion of value beyond that of ethics, but limited to Western sources\\nSocial imaginary is the set of values, institutions, laws, and symbols common to a particular social group\\nValue (economics), a measure of the benefit that may be gained from goods or service\\nTheory of value (economics), the study of the concept of economic value\\nValue (marketing), the difference between a customer\\'s evaluation of benefits and costs\\nValue investing, an investment paradigm\\n\\nOther uses\\nValue, also known as lightness or tone, a representation of variation in the perception of a color or color space\\'s brightness\\nValue (computer science), an expression that implies no (further) (mathematical) processing; a \"normal form\"\\nValue (mathematics), a property such as number assigned to or calculated for a variable, constant or expression\\nValue (semiotics), the significance, purpose and/or meaning of a symbol as determined or affected by other symbols\\nNote value, the relative duration of a musical note\\nValues (political party), a defunct New Zealand environmentalist political party\\n\\nSee also\\nInstrumental and intrinsic value\\nValue theory, a range of approaches to understanding how, why, and to what degree people value things'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "give_harder_level(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Model\n",
    "The final model functions can be found in [functions.py](functions.py). The difference between model 4 and the current model delopyed on the web app is primarily in functionality. There is no difference in the calculation in cosine distance between article vectors between M4 and M5. M4 does give a grade to the user input text while M5 does not. M5's functions do have different outputs to deliver a catalog of articles rather than one article. This change was made to improve the user experience of the web app. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}