NLP_Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# text mining (nlp) with python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** Ties de Kok   ([Personal Website](https://www.tiesdekok.com))   <br>\n",
    "**Last updated:** June 2020  \n",
    "**Python version:** Python 3.7  \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Some features (like the ToC) will only work if you run the notebook or if you use nbviewer by clicking this link:  \n",
    "https://nbviewer.jupyter.org/github/TiesdeKok/Python_NLP_Tutorial/blob/master/NLP_Notebook.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains code examples to get you started with Python for Natural Language Processing (NLP) / Text Mining.  \n",
    "\n",
    "In the large scheme of things there are roughly 4 steps:  \n",
    "\n",
    "1. Identify a data source  \n",
    "2. Gather the data  \n",
    "3. Process the data  \n",
    "4. Analyze the data  \n",
    "\n",
    "This notebook only discusses step 3 and 4. If you want to learn more about step 2 see my [Python tutorial](https://github.com/TiesdeKok/LearnPythonforResearch). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note: companion slides"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook was designed to accompany a PhD course session on NLP techniques in Accounting Research.  \n",
    "The slides of this session are publically availabe here: [Slides](http://www.tiesdekok.com/AccountingNLP_Slides/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Elements / topics that are discussed in this notebook:*\n",
    "\n",
    "\n",
    "<img style=\"float: left\" src=\"https://i.imgur.com/c3aCZLA.png\" width=\"50%\" /> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Table of Contents*  <a id='toc'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Primer on NLP tools](#tool_primer)     \n",
    "* [Process + Clean text](#proc_clean)   \n",
    "    * [Normalization](#normalization)\n",
    "        * [Deal with unwanted characters](#unwanted_char)\n",
    "        * [Sentence segmentation](#sentence_seg)   \n",
    "        * [Word tokenization](#word_token)\n",
    "        * [Lemmatization & Stemming](#lem_and_stem) \n",
    "    * [Language modeling](#lang_model) \n",
    "        * [Part-of-Speech tagging](#pos_tagging) \n",
    "        * [Uni-Gram & N-Grams](#n_grams) \n",
    "        * [Stop words](#stop_words) \n",
    "* [Direct feature extraction](#feature_extract) \n",
    "    * [Feature search](#feature_search) \n",
    "        * [Entity recognition](#entity_recognition) \n",
    "        * [Pattern search](#pattern_search) \n",
    "    * [Text evaluation](#text_eval) \n",
    "        * [Language](#language) \n",
    "        * [Dictionary counting](#dict_counting) \n",
    "        * [Readability](#readability) \n",
    "* [Represent text numerically](#text_numerical) \n",
    "    * [Bag of Words](#bows) \n",
    "        * [TF-IDF](#tfidf) \n",
    "    * [Word Embeddings](#word_embed) \n",
    "        * [Spacy](#spacyEmbedding)\n",
    "        * [Word2Vec](#Word2Vec) \n",
    "* [Statistical models](#stat_models) \n",
    "    * [\"Traditional\" machine learning](#trad_ml) \n",
    "        * [Supervised](#trad_ml_supervised) \n",
    "            * [Naïve Bayes](#trad_ml_supervised_nb) \n",
    "            * [Support Vector Machines (SVM)](#trad_ml_supervised_svm) \n",
    "        * [Unsupervised](#trad_ml_unsupervised) \n",
    "            * [Latent Dirichilet Allocation (LDA)](#trad_ml_unsupervised_lda) \n",
    "            * [pyLDAvis](#trad_ml_unsupervised_pyLDAvis) \n",
    "* [Model Selection and Evaluation](#trad_ml_eval) \n",
    "* [Neural Networks](#nn_ml)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Primer on NLP tools</span><a id='tool_primer'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many tools available for NLP purposes.  \n",
    "The code examples below are based on what I personally like to use, it is not intended to be a comprehsnive overview.  \n",
    "\n",
    "Besides build-in Python functionality I will use / demonstrate the following packages:\n",
    "\n",
    "**Standard NLP libraries**:\n",
    "1. `Spacy` \n",
    "2. `NLTK` and the higher-level wrapper `TextBlob`\n",
    "\n",
    "*Note: besides installing the above packages you also often have to download (model) data . Make sure to check the documentation!*\n",
    "\n",
    "**Standard machine learning library**:\n",
    "\n",
    "1. `scikit learn`\n",
    "\n",
    "**Specific task libraries**:\n",
    "\n",
    "There are many, just a couple of examples:\n",
    "\n",
    "1. `pyLDAvis` for visualizing LDA)\n",
    "2. `langdetect` for detecting languages\n",
    "3. `fuzzywuzzy` for fuzzy text matching\n",
    "4. `Gensim` for topic modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Get some example data</span><a id='example_data'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many example datasets available to play around with, see for example this great repository:  \n",
    "https://archive.ics.uci.edu/ml/datasets.php"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data that I will use for most of the examples is the \"Reuter_50_50 Data Set\" that is used for author identification experiments. \n",
    "\n",
    "See the details here: https://archive.ics.uci.edu/ml/datasets/Reuter_50_50  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download and load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can't follow what I am doing here? Please see my [Python tutorial](https://github.com/TiesdeKok/LearnPythonforResearch) (although the `zipfile` and `io` operations are not very relevant)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, zipfile, io, os\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* for `tqdm` to work in JupyterLab you need to install the `@jupyter-widgets/jupyterlab-manager` using the puzzle icon in the left side bar. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Download and extract the zip file with the data *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('C50test'):\n",
    "    r = requests.get(\"https://archive.ics.uci.edu/ml/machine-learning-databases/00217/C50.zip\")\n",
    "    z = zipfile.ZipFile(io.BytesIO(r.content))\n",
    "    z.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Load the data into memory*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_dict = {'test' : 'C50test'}\n",
    "text_dict = {'test' : {}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "579743db1d6e4759ba60efb6b991da59",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for label, folder in tqdm(folder_dict.items()):\n",
    "    authors = os.listdir(folder)\n",
    "    for author in authors:\n",
    "        text_files = os.listdir(os.path.join(folder, author))\n",
    "        for file in text_files:\n",
    "            with open(os.path.join(folder, author, file), 'r') as text_file:\n",
    "                text_dict[label].setdefault(author, []).append(' '.join(text_file.readlines()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: the text comes pre-split per sentence, for the sake of example I undo this through `' '.join(text_file.readlines()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain\\'s Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\\n Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers Commission which is due to report before March 24, 1997. The shares fell 6p to 781p on the news.\\n \"The stock is probably dead in the water until March,\" said John Wakley, analyst at Lehman Brothers.  \\n Dermott Carr, an analyst at Nikko said, \"the market is going to hang onto them for the moment but until we get a decision they will be held back.\"\\n Whatever the MMC decides many analysts expect Lang to defer a decision until after the next general election which will be called by May 22.\\n \"They will probably try to defer the decision until after the election. I don\\'t think they want the negative PR of having a large number of people fired,\" said Wakley.  \\n If the deal does not go through, analysts calculate the maximum loss to Bass of 60 million, with most sums centred on the 30-40 million range.\\n \"It\\'s a maxiumum loss of 60 million for Bass if they fail and, unlike Allied, you would have to compare it to the perceived upside of doing the deal,\" said Wakley.\\n Bass said at the time of the deal it would take a one-off charge of 75 million stg for restructuring the combined business, resulting in expected annual cost savings of 90 million stg within three years.  \\n Under the terms of the complex deal, if Bass cannot combine C-T with its own brewing business within 16 months, it has the option to put its whole shareholding to Carlsberg for 110 million stg and Carlsberg has an option to put 15 percent of C-T to Allied Domecq, which would reimburse Bass 30 million stg.\\n Bass is also entitled to receive 50 percent of all profits earnied by C-T until the merger is complete, which should give it some 30-35 million stg in a full year. Carlsberg has agreed to contribute its interests and 20 million stg in exchange for a 20 percent share in the combined Bass Breweries and Carlsberg-Tetley business.\\n C-T was a joint venture between Allied Domecq and Carlsberg formed in 1992 by the merger of their UK brewing and wholesaleing businesses.\\n -- London Newsroom +44 171 542 6437\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_dict['test']['TimFarrand'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Process + Clean text</span><a id='proc_clean'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the text into a NLP representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the text directly, but if want to use packages like `spacy` and `textblob` we first have to convert the text into a corresponding object.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** depending on the way that you installed the language models you will need to import it differently:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "from spacy.en import English\n",
    "nlp = English()\n",
    "```\n",
    "OR\n",
    "```\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "import en_core_web_md\n",
    "nlp = en_core_web_md.load()\n",
    "\n",
    "import en_core_web_lg\n",
    "nlp = en_core_web_lg.load()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import en_core_web_md\n",
    "nlp = en_core_web_md.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert all text in the \"test\" sample to a `spacy` `doc` object using `nlp.pipe()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7819909a0ff74c40b2201cbd78c04984",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "spacy_text = {}\n",
    "for author, text_list in tqdm(text_dict['test'].items()):\n",
    "    spacy_text[author] = list(nlp.pipe(text_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*A note on speed:*  This is slow because we didn't disable any compontents, see this note from the documentation:  \n",
    "> Only apply the pipeline components you need. Getting predictions from the model that you don’t actually need adds up and becomes very inefficient at scale. To prevent this, use the disable keyword argument to disable components you don’t need – either when loading a model, or during processing with nlp.pipe. See the section on disabling pipeline components for more details and examples. [link](https://spacy.io/usage/processing-pipelines#disabling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spacy.tokens.doc.Doc"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(spacy_text['TimFarrand'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can apply basic `nltk` operations directly to the text so we don't need to convert first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert all text in the \"test\" sample to a `TextBlob` object using `TextBlob()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "textblob_text = {}\n",
    "for author, text_list in text_dict['test'].items():\n",
    "    textblob_text[author] = [TextBlob(text) for text in text_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "textblob.blob.TextBlob"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(textblob_text['TimFarrand'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Normalization</span><a id='normalization'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Text normalization** describes the task of transforming the text into a different (more comparable) form.  \n",
    "\n",
    "This can imply many things, I will show a couple of options below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Deal with unwanted characters</span><a id='unwanted_char'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will often notice that there are characters that you don't want in your text.  \n",
    "\n",
    "Let's look at this sentence for example:\n",
    "\n",
    "> \"Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain\\'s Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\\n Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers\"\n",
    "\n",
    "You notice that there are some `\\` and `\\n` in there. These are used to define how a string should be displayed, if we print this text we get:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\\n Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers\""
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_dict['test']['TimFarrand'][0][:298]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\n",
      " Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers\n"
     ]
    }
   ],
   "source": [
    "print(text_dict['test']['TimFarrand'][0][:298])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These special characters can cause problems in our analyses (and can be hard to debug if you are using `print` statements to inspect the data).\n",
    "\n",
    "**So how do we remove them?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases it is sufficient to simply use the `.replace()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts. Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_dict['test']['TimFarrand'][0][:298].replace('\\n', '').replace('\\\\', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, however, the problem arrises because of encoding / decoding problems.  \n",
    "\n",
    "In those cases you can usually do something like:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is some π text that has to be cleaned…! it's difficult to deal with!\n",
      "b\"This is some  text that has to be cleaned! it's difficult to deal with!\"\n"
     ]
    }
   ],
   "source": [
    "problem_sentence = 'This is some \\u03c0 text that has to be cleaned\\u2026! it\\u0027s difficult to deal with!'\n",
    "print(problem_sentence)\n",
    "print(problem_sentence.encode().decode('unicode_escape').encode('ascii','ignore'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative that is better at preserving the unicode characters would be to use `unidecode`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unidecode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "王玉\n"
     ]
    }
   ],
   "source": [
    "print('\\u738b\\u7389')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Wang Yu '"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unidecode.unidecode(u\"\\u738b\\u7389\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"This is some p text that has to be cleaned...! it's difficult to deal with!\""
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unidecode.unidecode(problem_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Sentence segmentation</span><a id='sentence_seg'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentence segmentation refers to the task of splitting up the text by sentence.  \n",
    "\n",
    "You could do this by splitting on the `.` symbol, but dots are used in many other cases as well so it is not very robust:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts\",\n",
       " '\\n Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers Commission which is due to report before March 24, 1997',\n",
       " ' The shares fell 6p to 781p on the news',\n",
       " '\\n \"The stock is probably dead in the water until March,\" said John Wakley, analyst at Lehman Brothers',\n",
       " '  \\n Dermott Carr, an analyst at Nikko said, \"the mark']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_dict['test']['TimFarrand'][0][:550].split('.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is better to use a more sophisticated implementation such as the one by `Spacy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_paragraph = spacy_text['TimFarrand'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\n",
       "  ,\n",
       " Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers Commission which is due to report before March 24, 1997.,\n",
       " The shares fell 6p to 781p on the news.\n",
       "  ,\n",
       " \"The stock is probably dead in the water until March,\" said John Wakley, analyst at Lehman Brothers.  \n",
       "  ,\n",
       " Dermott Carr, an analyst at Nikko said, \"the market is going to hang onto them for the moment]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_list = [s for s in example_paragraph.sents]\n",
    "sentence_list[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the returned object is still a `spacy` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spacy.tokens.span.Span"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(sentence_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `spacy` sentence segmentation relies on the text being capitalized, so make sure you didn't convert it to all lower case before running this operation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply to all texts (for use later on):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd91842ae5e54926b754e5be67f3567f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "spacy_sentences = {}\n",
    "for author, text_list in tqdm(spacy_text.items()):\n",
    "    spacy_sentences[author] = [list(text.sents) for text in text_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\n",
       "  ,\n",
       " Earlier Lang announced the Bass deal would be referred to the Monoplies and Mergers Commission which is due to report before March 24, 1997.,\n",
       " The shares fell 6p to 781p on the news.\n",
       "  ]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_sentences['TimFarrand'][0][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Word tokenization</span><a id='word_token'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word tokenization means to split the sentence (or text) up into words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Shares in brewing-to-leisure group Bass Plc are likely to be held back until Britain's Trade and Industry secretary Ian Lang decides whether to allow its proposed merge with brewer Carlsberg-Tetley, said analysts.\n",
       " "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_sentence = spacy_sentences['TimFarrand'][0][0]\n",
    "example_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A word is called a `token` in this context (hence `tokenization`), using `spacy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares,\n",
       " in,\n",
       " brewing,\n",
       " -,\n",
       " to,\n",
       " -,\n",
       " leisure,\n",
       " group,\n",
       " Bass,\n",
       " Plc,\n",
       " are,\n",
       " likely,\n",
       " to,\n",
       " be,\n",
       " held]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_list = [token for token in example_sentence]\n",
    "token_list[0:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Lemmatization & Stemming</span><a id='lem_and_stem'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases you want to convert a word (i.e. token) into a more general representation.  \n",
    "\n",
    "For example: convert \"car\", \"cars\", \"car's\", \"cars'\" all into the word `car`.\n",
    "\n",
    "This is generally done through lemmatization / stemming (different approaches trying to achieve a similar goal).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Spacy**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Space offers build-in functionality for lemmatization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['share',\n",
       " 'in',\n",
       " 'brewing',\n",
       " '-',\n",
       " 'to',\n",
       " '-',\n",
       " 'leisure',\n",
       " 'group',\n",
       " 'Bass',\n",
       " 'Plc',\n",
       " 'be',\n",
       " 'likely',\n",
       " 'to',\n",
       " 'be',\n",
       " 'hold']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lemmatized = [token.lemma_ for token in example_sentence]\n",
    "lemmatized[0:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NLTK**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the NLTK libary we can also use the more aggressive Porter Stemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['share',\n",
       " 'in',\n",
       " 'brew',\n",
       " '-',\n",
       " 'to',\n",
       " '-',\n",
       " 'leisur',\n",
       " 'group',\n",
       " 'bass',\n",
       " 'plc',\n",
       " 'are',\n",
       " 'like',\n",
       " 'to',\n",
       " 'be',\n",
       " 'held']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmed = [stemmer.stem(token.text) for token in example_sentence]\n",
    "stemmed[0:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Compare**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Original  | Spacy Lemma  | NLTK Stemmer\n",
      "-----------------------------------------\n",
      "    Shares  |       share  |       share\n",
      "        in  |          in  |          in\n",
      "   brewing  |     brewing  |        brew\n",
      "         -  |           -  |           -\n",
      "        to  |          to  |          to\n",
      "         -  |           -  |           -\n",
      "   leisure  |     leisure  |      leisur\n",
      "     group  |       group  |       group\n",
      "      Bass  |        Bass  |        bass\n",
      "       Plc  |         Plc  |         plc\n",
      "       are  |          be  |         are\n",
      "    likely  |      likely  |        like\n",
      "        to  |          to  |          to\n",
      "        be  |          be  |          be\n",
      "      held  |        hold  |        held\n"
     ]
    }
   ],
   "source": [
    "print('  Original  | Spacy Lemma  | NLTK Stemmer')\n",
    "print('-' * 41)\n",
    "for original, lemma, stem in zip(token_list[:15], lemmatized[:15], stemmed[:15]):\n",
    "    print(str(original).rjust(10, ' '), ' | ', str(lemma).rjust(10, ' '), ' | ', str(stem).rjust(10, ' '))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my experience it is usually best to use lemmatization instead of a stemmer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Language modeling</span><a id='lang_model'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text is inherently structured in complex ways, we can often use some of this underlying structure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Part-of-Speech tagging</span><a id='pos_tagging'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part of speech tagging refers to the identification of words as nouns, verbs, adjectives, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `Spacy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Shares, 'NOUN'),\n",
       " (in, 'ADP'),\n",
       " (brewing, 'NOUN'),\n",
       " (-, 'PUNCT'),\n",
       " (to, 'ADP'),\n",
       " (-, 'PUNCT'),\n",
       " (leisure, 'NOUN'),\n",
       " (group, 'NOUN'),\n",
       " (Bass, 'PROPN'),\n",
       " (Plc, 'PROPN')]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_list = [(token, token.pos_) for token in example_sentence]\n",
    "pos_list[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Uni-Gram & N-Grams</span><a id='n_grams'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously a sentence is not a random collection of words, the sequence of words has information value.  \n",
    "\n",
    "A simple way to incorporate some of this sequence is by using what is called `n-grams`.  \n",
    "An `n-gram` is nothing more than a a combination of `N` words into one token (a uni-gram token is just one word).  \n",
    "\n",
    "So we can convert `\"Sentence about flying cars\"` into a list of bigrams:\n",
    "\n",
    "> Sentence-about, about-flying, flying-cars  \n",
    "\n",
    "See my slide on N-Grams for a more comprehensive example: [click here](http://www.tiesdekok.com/AccountingNLP_Slides/#14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `NLTK`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['are-likely', 'likely-to', 'to-be', 'be-held', 'held-back']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_list = ['-'.join(x) for x in nltk.bigrams([token.text for token in example_sentence])]\n",
    "bigram_list[10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `spacy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_without_punctuation(sen_obj):\n",
    "    return [token.text for token in sen_obj if token.is_alpha]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_ngram(sen_obj, n, sep = '-'):\n",
    "    token_list = tokenize_without_punctuation(sen_obj)\n",
    "    number_of_tokens = len(token_list)\n",
    "    ngram_list = []\n",
    "    for i, token in enumerate(token_list[:-n+1]):\n",
    "        ngram_item = [token_list[i + ii] for ii in range(n)]\n",
    "        ngram_list.append(sep.join(ngram_item))\n",
    "    return ngram_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Shares-in', 'in-brewing', 'brewing-to', 'to-leisure', 'leisure-group']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_ngram(example_sentence, 2)[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Shares-in-brewing',\n",
       " 'in-brewing-to',\n",
       " 'brewing-to-leisure',\n",
       " 'to-leisure-group',\n",
       " 'leisure-group-Bass']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_ngram(example_sentence, 3)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Stop words</span><a id='stop_words'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on what you are trying to do it is possible that there are many words that don't add any information value to the sentence.  \n",
    "\n",
    "The primary example are stop words.  \n",
    "\n",
    "Sometimes you can improve the accuracy of your model by removing stop words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `Spacy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "no_stop_words = [token for token in example_sentence if not token.is_stop]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares, brewing, -, -, leisure, group, Bass, Plc, likely, held]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "no_stop_words[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares, in, brewing, -, to, -, leisure, group, Bass, Plc]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_list[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note* we can also remove punctuation in the same way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shares, brewing, leisure, group, Bass, Plc, likely, held, Britain, Trade]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[token for token in example_sentence if not token.is_stop and token.is_alpha][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap everything into one function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic SpaCy text processing function**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Split into sentences\n",
    "2. Apply lemmatizer, remove top words, remove punctuation\n",
    "3. Clean up the sentence using `textacy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_text_custom(text):\n",
    "    sentences = list(nlp(text, disable=['tagger', 'ner', 'entity_linker', 'textcat', 'entitry_ruler']).sents)\n",
    "    lemmatized_sentences = []\n",
    "    for sentence in sentences:\n",
    "        lemmatized_sentences.append([token.lemma_ for token in sentence if not token.is_stop and token.is_alpha])\n",
    "    return [' '.join(sentence) for sentence in lemmatized_sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1daf8d39277d48adb85050ff35d9101e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "spacy_text_clean = {}\n",
    "for author, text_list in tqdm(text_dict['test'].items()):\n",
    "    lst = []\n",
    "    for text in text_list:\n",
    "        lst.append(process_text_custom(text))\n",
    "    spacy_text_clean[author] = lst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* that this would take quite a long time if we didn't disable some of the components. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences: 58973\n"
     ]
    }
   ],
   "source": [
    "count = 0\n",
    "for author, texts in spacy_text_clean.items():\n",
    "    for text in texts:\n",
    "        count += len(text)\n",
    "print('Number of sentences:', count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Shares brew leisure group Bass Plc likely hold Britain Trade Industry secretary Ian Lang decide allow propose merge brewer Carlsberg Tetley say analyst',\n",
       " 'Earlier Lang announce Bass deal refer Monoplies Mergers Commission report March',\n",
       " 'share fall news']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_text_clean['TimFarrand'][0][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* the quality of the input text is not great, so the sentence segmentation is also not great (without further tweaking)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Direct feature extraction</span><a id='feature_extract'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have pre-processed our text into something that we can use for direct feature extraction or to convert it to a numerical representation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Feature search</span><a id='feature_search'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Entity recognition</span><a id='entity_recognition'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is often useful / relevant to extract entities that are mentioned in a piece of text.   \n",
    "\n",
    "SpaCy is quite powerful in extracting entities, however, it doesn't work very well on lowercase text.  \n",
    "\n",
    "Given that \"token.lemma\\_\" removes capitalization I will use `spacy_sentences` for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The stock is probably dead in the water until March,\" said John Wakley, analyst at Lehman Brothers.  \n",
       " "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_sentence = spacy_sentences['TimFarrand'][0][3]\n",
    "example_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(March, 'DATE'), (John Wakley, 'PERSON'), (Lehman Brothers, 'ORG')]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(i, i.label_) for i in nlp(example_sentence.text).ents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "British pub-to-hotel group Greenalls Plc on Thursday reported a 48 percent rise in profits before exceptional items to 148.7 million pounds ($246.4 million), driven by its acquisition of brewer Boddington in November 1995.\n",
       " "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_sentence = spacy_sentences['TimFarrand'][4][0]\n",
    "example_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(British, 'NORP'),\n",
       " (Greenalls Plc, 'ORG'),\n",
       " (Thursday, 'DATE'),\n",
       " (48 percent, 'PERCENT'),\n",
       " (148.7 million pounds, 'MONEY'),\n",
       " ($246.4 million, 'MONEY'),\n",
       " (Boddington, 'ORG'),\n",
       " (November 1995, 'DATE')]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(i, i.label_) for i in nlp(example_sentence.text).ents]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Pattern search</span><a id='pattern_search'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the build-in `re` (regular expression) library you can pattern match nearly anything you want.  \n",
    "\n",
    "I will not go into details about regular expressions but see here for a tutorial:  \n",
    "https://regexone.com/references/python  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TIP**: Use [Pythex.org](https://pythex.org/) to try out your regular expression\n",
    "\n",
    "Example on Pythex: <a href=\"https://pythex.org/?regex=IDNUMBER: (\\d\\d\\d-\\w\\w)&test_string=Ties de Kok (IDNUMBER: 123-AZ). Rest of Text.\" target='_blank'>click here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example 1:**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "string_1 = 'Ties de Kok (#IDNUMBER: 123-AZ). Rest of text...'\n",
    "string_2 = 'Philip Joos (#IDNUMBER: 663-BY). Rest of text...'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = r'#IDNUMBER: (\\d\\d\\d-\\w\\w)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "123-AZ\n",
      "663-BY\n"
     ]
    }
   ],
   "source": [
    "print(re.findall(pattern, string_1)[0])\n",
    "print(re.findall(pattern, string_2)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a sentence contains the word 'million' return True, otherwise return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analysts forecast pretax profit range million stg restructure cost million time\n",
      "restructure cost million anticipate bulk million stem closure small production plant France\n",
      "Cadbury drink business turn million stg trade profit million half entirely contribution Dr Pepper\n",
      "Campbell estimate UK beverage contribute million stg operate profit million time\n",
      "Broadly analyst expect pretty flat performance group confectionery business consensus forecast million stg operate profit\n",
      "average analyst calculate beverage chip trade profit million\n",
      "sale percent stake Coca Cola amp Schweppes Beverages CCSB operation Coca Cola Enterprises June million stg analyst want clear statement strategy company\n",
      "far analyst company say shareholder expect return investment emerge market large far million Russian plant\n",
      "Cadbury announce investment million stg build new plant Wrocoaw Poland joint venture China cost million\n",
      "Net debt billion end fall million end result CCSB sale provide acquisition\n"
     ]
    }
   ],
   "source": [
    "for sen in spacy_text_clean['TimFarrand'][2]:\n",
    "    TERM = 'million'\n",
    "    if re.search('million', sen, flags= re.IGNORECASE):\n",
    "        print(sen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Text evaluation</span><a id='text_eval'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides feature search there are also many ways to analyze the text as a whole.  \n",
    "\n",
    "Let's, for example, evaluate the following paragraph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Soft drink confectionery group Cadbury Schweppes Plc expect report solid percent rise half profit Wednesday face question performance soft drink main question success relaunch brand say Mark Duffy food manufacture analyst SBC Warburg Competitor Sprite own Coca Cola see agressive market push rank fast grow brand Cadbury Dr Pepper Analysts forecast pretax profit range million stg restructure cost million time dividend penny expect restructure cost million anticipate bulk million stem closure small'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_paragraph = ' '.join([x for x in spacy_text_clean['TimFarrand'][2]])\n",
    "example_paragraph[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Language</span><a id='language'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `spacy-langdetect` package it is easy to detect the language of a piece of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy_langdetect import LanguageDetector\n",
    "nlp.add_pipe(LanguageDetector(), name='language_detector', last=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'language': 'en', 'score': 0.9999970401265338}\n"
     ]
    }
   ],
   "source": [
    "print(nlp(example_paragraph)._.language)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Readability</span><a id='readability'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally I'd recommend to calculate the readability metrics by yourself as they don't tend to be that difficult to compute. However, there are packages out there that can help, such as `spacy_readability`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy_readability import Readability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.add_pipe(Readability(), name='readability', last=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.412857142857145\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I am some really difficult text to read because I use obnoxiously large words.\")\n",
    "print(doc._.flesch_kincaid_grade_level)\n",
    "print(doc._.smog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Manual example:** FOG index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "import syllapy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_fog(document):\n",
    "    doc = nlp(document, disable=['tagger', 'ner', 'entity_linker', 'textcat', 'entitry_ruler'])\n",
    "    sen_list = list(doc.sents)\n",
    "    num_sen = len(sen_list)\n",
    "\n",
    "    num_words = 0\n",
    "    num_complex_words = 0\n",
    "    for sen_obj in sen_list:\n",
    "        words_in_sen = [token.text for token in sen_obj if token.is_alpha]\n",
    "        num_words += len(words_in_sen)\n",
    "        num_complex  = 0\n",
    "        for word in words_in_sen:\n",
    "            num_syl = syllapy.count(word.lower())\n",
    "            if num_syl > 2:\n",
    "                num_complex += 1\n",
    "        num_complex_words += num_complex\n",
    "        \n",
    "    fog = 0.4 * ((num_words / num_sen) + ((num_complex_words / num_words)*100))\n",
    "    return {'fog' : fog, \n",
    "            'num_sen' : num_sen, \n",
    "            'num_words' : num_words, \n",
    "            'num_complex_words' : num_complex_words}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fog': 13.42327889849504,\n",
       " 'num_sen': 36,\n",
       " 'num_words': 347,\n",
       " 'num_complex_words': 83}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_fog(example_paragraph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `fuzzywuzzy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fuzzywuzzy import fuzz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fuzz.ratio(\"fuzzy wuzzy was a bear\", \"wuzzy fuzzy was a bear\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `spacy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spacy can provide a similary score based on the semantic similarity ([link](https://spacy.io/usage/vectors-similarity))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0000000623731768"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_1 = nlp(\"fuzzy wuzzy was a bear\")\n",
    "tokens_2 = nlp(\"wuzzy fuzzy was a bear\")\n",
    "\n",
    "tokens_1.similarity(tokens_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8127869114665882"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_1 = nlp(\"Tom believes German cars are the best.\")\n",
    "tokens_2 = nlp(\"Sarah recently mentioned that she would like to go on holiday to Germany.\")\n",
    "\n",
    "tokens_1.similarity(tokens_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Term (dictionary) counting</span><a id='dict_counting'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common technique for basic NLP insights is to create simple metrics based on term counts. \n",
    "\n",
    "These are relatively easy to implement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_dictionary = ['soft', 'first', 'most', 'be']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "soft 2\n",
      "first 0\n",
      "most 0\n",
      "be 7\n"
     ]
    }
   ],
   "source": [
    "for word in word_dictionary:\n",
    "    print(word, example_paragraph.count(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos = ['great', 'agree', 'increase']\n",
    "neg = ['bad', 'disagree', 'decrease']\n",
    "\n",
    "sentence = '''According to the president everything is great, great, \n",
    "and great even though some people might disagree with those statements.'''\n",
    "\n",
    "pos_count = 0\n",
    "for word in pos:\n",
    "    pos_count += sentence.lower().count(word)\n",
    "print(pos_count)\n",
    "\n",
    "neg_count = 0\n",
    "for word in neg:\n",
    "    neg_count += sentence.lower().count(word)\n",
    "print(neg_count)\n",
    "\n",
    "pos_count / (neg_count + pos_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting the total number of words is also easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_tokens = len([token for token in nlp(sentence) if token.is_alpha])\n",
    "num_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also save the count per word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_count_dict = {}\n",
    "for word in pos:\n",
    "    pos_count_dict[word] = sentence.lower().count(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'great': 3, 'agree': 1, 'increase': 0}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_count_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `.lower()` is actually quite slow, if you have a lot of words / sentences it is recommend to minimize the amount of `.lower()` operations that you have to make."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Represent text numerically</span><a id='text_numerical'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Bag of Words</span><a id='bows'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sklearn includes the `CountVectorizer` and `TfidfVectorizer` function.  \n",
    "\n",
    "For details, see the documentation:  \n",
    "[TF](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer)  \n",
    "[TFIDF](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer)\n",
    "\n",
    "*Note 1:* these functions also provide a variety of built-in preprocessing options (e.g. ngrames, remove stop words, accent stripper).\n",
    "\n",
    "*Note 2:* example based on the following website [click here](http://ethen8181.github.io/machine-learning/clustering_old/tf_idf/tf_idf.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_1 = \"The sky is blue.\"\n",
    "doc_2 = \"The sun is bright today.\"\n",
    "doc_3 = \"The sun in the sky is bright.\"\n",
    "doc_4 = \"We can see the shining sun, the bright sun.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate term frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(stop_words='english')\n",
    "tf = vectorizer.fit_transform([doc_1, doc_2, doc_3, doc_4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['blue', 'bright', 'shining', 'sky', 'sun', 'today'] \n",
      "\n",
      "[1 0 0 1 0 0]\n",
      "[0 1 0 0 1 1]\n",
      "[0 1 0 1 1 0]\n",
      "[0 1 1 0 2 0]\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.get_feature_names(), '\\n')\n",
    "for doc_tf_vector in tf.toarray():\n",
    "    print(doc_tf_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">TF-IDF</span><a id='tfidf'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = TfidfVectorizer(stop_words='english')\n",
    "tfidf = transformer.fit_transform([doc_1, doc_2, doc_3, doc_4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.78528828 0.         0.         0.6191303  0.         0.        ]\n",
      "[0.         0.47380449 0.         0.         0.47380449 0.74230628]\n",
      "[0.         0.53256952 0.         0.65782931 0.53256952 0.        ]\n",
      "[0.         0.36626037 0.57381765 0.         0.73252075 0.        ]\n"
     ]
    }
   ],
   "source": [
    "for doc_vector in tfidf.toarray():\n",
    "    print(doc_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More elaborate example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_paragraphs = []\n",
    "for author, value in spacy_text_clean.items():\n",
    "    for article in value:\n",
    "        clean_paragraphs.append(' '.join([x for x in article]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2500"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(clean_paragraphs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = TfidfVectorizer(stop_words='english')\n",
    "tfidf_large = transformer.fit_transform(clean_paragraphs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of vectors: 2500\n",
      "Number of words in dictionary: 21978\n"
     ]
    }
   ],
   "source": [
    "print('Number of vectors:', len(tfidf_large.toarray()))\n",
    "print('Number of words in dictionary:', len(tfidf_large.toarray()[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2500x21978 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 410121 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_large"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Word Embeddings</span><a id='word_embed'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Spacy</span><a id='spacyEmbedding'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `en_core_web_lg` language model comes with GloVe vectors trained on the Common Crawl dataset ([link](https://spacy.io/models/en#en_core_web_lg))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The True 4.70935 False\n",
      "Dutch True 6.381229 False\n",
      "word True 5.8387117 False\n",
      "for True 4.8435082 False\n",
      "peanut True 7.085804 False\n",
      "butter True 7.466713 False\n",
      "is True 4.890306 False\n",
      "pindakaas False 0.0 False\n",
      "did True 5.284421 False\n",
      "you True 5.1979666 False\n",
      "know True 5.160699 False\n",
      "that True 4.8260193 False\n",
      "This True 5.0461264 False\n",
      "is True 4.890306 False\n",
      "a True 5.306696 False\n",
      "typpo False 0.0 True\n"
     ]
    }
   ],
   "source": [
    "tokens = nlp(\"The Dutch word for peanut butter is 'pindakaas', did you know that? This is a typpo.\")\n",
    "\n",
    "for token in tokens:\n",
    "    if token.is_alpha:\n",
    "        print(token.text, token.has_vector, token.vector_norm, token.is_oov)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The token: \"Car\" has the following vector (dimension: 300)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 2.0987e-01,  4.6481e-01, -2.4238e-01, -6.5751e-02,  6.0856e-01,\n",
       "       -3.4698e-01, -2.5331e-01, -4.2590e-01, -2.2277e-01,  2.2913e+00,\n",
       "       -3.3853e-01,  2.3275e-01, -2.7511e-01,  2.4064e-01, -1.0697e+00,\n",
       "       -2.6978e-01, -8.0733e-01,  1.8698e+00,  4.5562e-01, -1.4469e-01,\n",
       "        1.6246e-02, -3.5473e-01,  7.6152e-01, -6.8589e-02,  1.2156e-02,\n",
       "        9.0520e-03,  1.1131e-01, -3.0746e-01,  2.4168e-01,  1.1400e-01,\n",
       "        4.3952e-01, -6.6594e-01, -7.3198e-02,  8.0566e-01,  1.1748e-01,\n",
       "       -3.8758e-01,  1.0691e-01,  3.3697e-01, -1.3188e-01,  1.9364e-01,\n",
       "        5.5553e-01, -3.4029e-01,  1.7059e-01,  4.0736e-01, -1.6150e-01,\n",
       "        7.0302e-02,  6.7772e-02, -8.1763e-01,  3.0645e-01, -9.9862e-03,\n",
       "        9.4606e-02, -5.9763e-01,  1.4192e-01,  1.4857e-01, -3.1535e-01,\n",
       "        9.9092e-02,  2.0673e-01, -4.4041e-01,  2.1519e-01, -4.1294e-01,\n",
       "        2.6374e-01, -1.5493e-01,  2.4739e-01,  4.2090e-01,  1.8768e-01,\n",
       "        4.6904e-02,  9.6848e-02,  2.7431e-02,  1.0633e-01,  3.1926e-01,\n",
       "       -7.6260e-01, -8.8373e-02,  3.7519e-01,  4.7369e-01, -7.3557e-01,\n",
       "       -1.0760e-01, -2.6557e-02, -5.1079e-01, -1.8886e-01,  2.8679e-01,\n",
       "        6.5798e-02,  5.7129e-01,  2.5056e-01,  7.3858e-02,  8.4700e-03,\n",
       "        1.5158e-02,  7.3570e-01,  6.2549e-01,  5.1600e-02, -2.5802e-01,\n",
       "       -8.1203e-02,  1.3731e-01,  1.8809e-01, -6.5871e-01, -2.2361e-01,\n",
       "       -3.3318e-01,  1.5853e-01,  5.1523e-01,  5.0259e-01, -1.6894e-01,\n",
       "       -8.6465e-02,  2.5036e-01, -1.7419e-01, -2.7723e-02,  1.1262e-01,\n",
       "       -4.6449e-01,  1.6956e-01,  2.8931e-01, -1.3187e-01,  4.6368e-01,\n",
       "       -2.9348e-01, -3.1244e-01,  6.5886e-01, -4.7842e-01,  1.4754e-01,\n",
       "       -3.0646e-01,  4.3847e-01,  1.7684e-01, -1.1968e-01, -3.1002e-02,\n",
       "       -1.2228e-01, -5.6424e-01,  1.5289e-01, -7.9389e-01, -3.6731e-01,\n",
       "        1.6918e-01, -9.5210e-02,  1.6490e-01,  1.5936e-01,  1.2460e-01,\n",
       "        3.8846e-01,  2.3019e-01, -1.3054e-01, -2.1932e-01, -2.6782e-01,\n",
       "       -6.0745e-01, -3.4826e-01,  1.7656e-01, -1.0351e-01, -2.2750e-01,\n",
       "       -1.6111e+00, -4.0504e-01,  1.0872e+00, -1.6391e-01,  6.5586e-02,\n",
       "       -1.0632e-01, -1.4014e-01,  1.7712e-01,  7.1100e-01,  2.0313e-01,\n",
       "       -5.0138e-01,  1.7291e-01, -5.3208e-02, -4.0668e-01,  1.4907e-01,\n",
       "       -3.0631e-01,  4.6572e-01,  3.7977e-01, -1.3336e-01, -7.6937e-02,\n",
       "        1.1803e-02, -1.1185e-01,  7.0364e-01, -6.8615e-02,  5.8586e-01,\n",
       "       -5.9890e-01, -2.8104e-01,  4.9674e-01,  4.5867e-01,  1.6291e-01,\n",
       "       -2.8317e-01,  3.8870e-01, -3.9882e-01,  2.2407e-01, -2.3704e-01,\n",
       "       -2.3155e-01,  6.7882e-02,  8.3828e-01,  1.3231e-01,  2.9778e-01,\n",
       "        1.8471e-01, -9.7415e-05, -6.9993e-01,  4.6959e-03, -3.5461e-01,\n",
       "       -9.6413e-02,  1.0312e-01,  8.5293e-02, -2.6909e-01,  4.3886e-01,\n",
       "        3.1275e-01,  2.2829e-01,  4.8072e-01,  1.8399e-01, -1.7628e-01,\n",
       "       -4.8322e-01,  9.5676e-02, -2.4499e-01,  5.8915e-02, -3.9355e-02,\n",
       "       -4.6954e-01, -2.6272e-01,  1.5462e-01, -1.8055e-01,  1.6881e-03,\n",
       "        5.7027e-02, -6.7284e-02,  2.4853e-01,  3.5735e-01,  1.4325e-01,\n",
       "       -4.9276e-01, -2.9321e-02,  5.1167e-02,  4.9620e-01,  3.7308e-01,\n",
       "        4.0203e-01,  9.2905e-02,  7.4061e-01, -3.3765e-01, -3.5641e-01,\n",
       "        6.1675e-01, -9.5517e-01, -2.7492e-01,  2.2079e-01, -2.8898e-01,\n",
       "       -1.5504e-01, -3.1433e-01,  5.8383e-01, -2.6138e-02, -2.7755e-01,\n",
       "       -4.7184e-02,  1.0504e-01, -4.2419e-01, -1.6414e-01, -5.0711e-01,\n",
       "        4.2617e-01,  3.6889e-01,  4.3267e-01, -4.4480e-03,  5.6442e-01,\n",
       "       -3.0964e-02,  7.7629e-02,  2.2218e-01,  1.2818e-01, -1.6235e-01,\n",
       "       -2.2912e-01,  4.9174e-01, -5.1937e-01, -2.0793e-01, -3.6868e-01,\n",
       "       -5.5714e-01, -1.9930e-01,  2.9782e-01,  7.5921e-02, -3.9895e-01,\n",
       "        8.1692e-01, -1.0221e-01, -3.8049e-01,  1.9906e-01, -1.9875e-02,\n",
       "        7.3431e-02, -1.3882e-01,  2.7914e-01, -4.5367e-01,  2.9227e-01,\n",
       "       -5.5489e-01, -4.2121e-01,  5.5667e-01, -4.5230e-01, -1.1956e-01,\n",
       "        1.3504e-01, -2.3580e-01,  7.4221e-01, -2.7890e-01, -5.4580e-02,\n",
       "        3.1944e-01,  3.6717e-01,  1.3430e-01,  1.3629e-01, -9.7458e-02,\n",
       "       -6.0310e-01, -1.7762e-01,  2.5910e-01,  3.3150e-01,  2.2701e-01,\n",
       "        6.3664e-01,  1.5324e-01, -3.2894e-01, -3.6749e-01, -2.0328e-01,\n",
       "       -1.1924e+00, -4.6395e-01,  6.6984e-01, -4.9404e-01,  4.4154e-01,\n",
       "       -4.3699e-01,  2.3538e-01,  3.2135e-01,  2.6649e-01,  2.2438e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token = nlp('Car')\n",
    "print('The token: \"{}\" has the following vector (dimension: {})'.format(token.text, len(token.vector)))\n",
    "token.vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"text-decoration: underline;\">Word2Vec</span><a id='Word2Vec'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple example below is from:  https://medium.com/@mishra.thedeepak/word2vec-in-minutes-gensim-nlp-python-6940f4e00980"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* you might have to run `nltk.download('brown')` to install the NLTK corpus files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from nltk.corpus import brown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = brown.sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = gensim.models.Word2Vec(sentences, min_count=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('brown_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = gensim.models.Word2Vec.load('brown_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find words most similar to 'mother':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('father', 0.9847322702407837), ('husband', 0.9656894207000732), ('wife', 0.9496790170669556), ('friend', 0.9323333501815796), ('son', 0.9279097318649292), ('nickname', 0.9207977652549744), ('eagle', 0.9097722768783569), ('addiction', 0.9071668982505798), ('voice', 0.9051918983459473), ('patient', 0.8966456055641174)]\n"
     ]
    }
   ],
   "source": [
    "print(model.wv.most_similar(\"mother\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the odd one out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cereal\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\envs\\limpergPython\\lib\\site-packages\\gensim\\models\\keyedvectors.py:877: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
      "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
     ]
    }
   ],
   "source": [
    "print(model.wv.doesnt_match(\"breakfast cereal dinner lunch\".split()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "garden\n"
     ]
    }
   ],
   "source": [
    "print(model.wv.doesnt_match(\"pizza pasta garden fries\".split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve vector representation of the word \"human\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.40230748, -0.36424252,  0.48136342,  0.37950972,  0.25889766,\n",
       "       -0.10284429,  0.2539682 , -0.557558  , -1.0109891 , -0.30390584,\n",
       "        0.02308058, -0.9227236 ,  0.06572528, -0.24571045, -0.17627639,\n",
       "        0.35974783, -0.29690138, -0.25977215,  0.465111  , -1.5026555 ,\n",
       "        0.23448153, -0.79958564, -0.6682266 , -0.51277363, -0.11112369,\n",
       "       -1.4914587 , -0.30484447,  1.3466982 , -0.45936054,  0.02780625,\n",
       "        0.31517667, -0.12471037,  0.46333146, -0.29451668,  0.28516975,\n",
       "        1.3195679 ,  0.02986159,  0.27836317, -0.5356812 , -0.5574794 ,\n",
       "        0.55741835, -0.3692916 ,  0.3067411 , -0.62016165,  0.6085465 ,\n",
       "        0.6336735 ,  0.9925447 , -0.2553504 , -0.3593044 , -0.29228973,\n",
       "       -0.05774796, -0.22645272, -0.594325  , -0.19128117,  0.13758877,\n",
       "        0.58251387, -0.12266693, -0.33289537, -0.81493866,  0.64220285,\n",
       "       -0.40921453,  1.7995448 ,  0.98320687,  0.66162825, -0.03371862,\n",
       "        0.30391327,  0.30519032, -0.02499808,  0.46001107, -0.5412774 ,\n",
       "       -0.14508785,  0.47390515, -0.01815019,  0.39801887,  0.33498788,\n",
       "       -0.70357895,  0.80516887,  0.08044272, -0.70585257, -0.7256744 ,\n",
       "       -0.95714486, -0.12571876, -0.20877206, -0.456315  ,  0.7478423 ,\n",
       "       -0.25637153,  0.78873783,  0.24834621,  0.4455648 ,  0.1293853 ,\n",
       "        0.17152755, -0.30077967,  0.5803442 ,  0.16445744,  0.60369337,\n",
       "       -0.2301575 , -0.19547687, -0.36981392,  0.23723377,  0.24412107],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv['human']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Statistical models</span><a id='stat_models'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">\"Traditional\" machine learning</span><a id='trad_ml'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The library to use for machine learning is scikit-learn ([\"sklearn\"](http://scikit-learn.org/stable/index.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span>Supervised</span><a id='trad_ml_supervised'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score, KFold, train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn import metrics\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the data into a pandas dataframe (so that we can input it easier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "article_list = []\n",
    "for author, value in spacy_text_clean.items():\n",
    "    for article in value:\n",
    "        article_list.append((author, ' '.join([x for x in article])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "article_df = pd.DataFrame(article_list, columns=['author', 'text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1325</th>\n",
       "      <td>LydiaZajc</td>\n",
       "      <td>Canadian unit Wal Mart Stores Inc bow Canadian...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1516</th>\n",
       "      <td>MarkBendeich</td>\n",
       "      <td>Australia labour market watchdog expect award ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2121</th>\n",
       "      <td>SarahDavison</td>\n",
       "      <td>Temperatures rise Hong Kong prepare midsummer ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2226</th>\n",
       "      <td>SimonCowell</td>\n",
       "      <td>British composite insurer Commercial Union Plc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1914</th>\n",
       "      <td>PierreTran</td>\n",
       "      <td>French defence electronic firm Thomson CSF soo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            author                                               text\n",
       "1325     LydiaZajc  Canadian unit Wal Mart Stores Inc bow Canadian...\n",
       "1516  MarkBendeich  Australia labour market watchdog expect award ...\n",
       "2121  SarahDavison  Temperatures rise Hong Kong prepare midsummer ...\n",
       "2226   SimonCowell  British composite insurer Commercial Union Plc...\n",
       "1914    PierreTran  French defence electronic firm Thomson CSF soo..."
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the sample into a training and test sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(article_df.text, article_df.author, test_size=0.20, random_state=3561)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 500\n"
     ]
    }
   ],
   "source": [
    "print(len(X_train), len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple function to train (i.e. fit) and evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_and_evaluate(clf, X_train, X_test, y_train, y_test):\n",
    "    \n",
    "    clf.fit(X_train, y_train)\n",
    "    \n",
    "    print(\"Accuracy on training set:\")\n",
    "    print(clf.score(X_train, y_train))\n",
    "    print(\"Accuracy on testing set:\")\n",
    "    print(clf.score(X_test, y_test))\n",
    "    \n",
    "    y_pred = clf.predict(X_test)\n",
    "    \n",
    "    print(\"Classification Report:\")\n",
    "    print(metrics.classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span>Naïve Bayes estimator</span><a id='trad_ml_supervised_nb'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = Pipeline([\n",
    "    ('vect', TfidfVectorizer(strip_accents='unicode',\n",
    "                             lowercase = True,\n",
    "                            max_features = 1500,\n",
    "                            stop_words='english'\n",
    "                            )),\n",
    "        \n",
    "    ('clf', MultinomialNB(alpha = 1,\n",
    "                          fit_prior = True\n",
    "                          )\n",
    "    ),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train and show evaluation stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training set:\n",
      "0.8345\n",
      "Accuracy on testing set:\n",
      "0.72\n",
      "Classification Report:\n",
      "                   precision    recall  f1-score   support\n",
      "\n",
      "    AaronPressman       0.82      1.00      0.90         9\n",
      "       AlanCrosby       0.55      0.92      0.69        12\n",
      "   AlexanderSmith       0.86      0.60      0.71        10\n",
      "  BenjaminKangLim       0.75      0.27      0.40        11\n",
      "    BernardHickey       0.75      0.30      0.43        10\n",
      "      BradDorfman       0.80      1.00      0.89         8\n",
      " DarrenSchuettler       0.58      0.78      0.67         9\n",
      "      DavidLawder       1.00      0.60      0.75        10\n",
      "    EdnaFernandes       1.00      0.67      0.80         9\n",
      "      EricAuchard       0.86      0.67      0.75         9\n",
      "   FumikoFujisaki       1.00      1.00      1.00        10\n",
      "   GrahamEarnshaw       0.59      1.00      0.74        10\n",
      " HeatherScoffield       0.83      0.56      0.67         9\n",
      "       JanLopatka       0.38      0.33      0.35         9\n",
      "    JaneMacartney       0.33      0.60      0.43        10\n",
      "     JimGilchrist       0.73      1.00      0.84         8\n",
      "   JoWinterbottom       0.90      0.90      0.90        10\n",
      "         JoeOrtiz       0.80      0.89      0.84         9\n",
      "     JohnMastrini       0.83      0.29      0.43        17\n",
      "     JonathanBirt       0.53      1.00      0.70         8\n",
      "      KarlPenhaul       0.87      1.00      0.93        13\n",
      "        KeithWeir       0.69      0.90      0.78        10\n",
      "   KevinDrawbaugh       0.88      0.70      0.78        10\n",
      "    KevinMorrison       0.33      1.00      0.50         3\n",
      "    KirstinRidley       0.86      0.67      0.75         9\n",
      "KouroshKarimkhany       0.54      0.88      0.67         8\n",
      "        LydiaZajc       0.90      0.90      0.90        10\n",
      "   LynneO'Donnell       0.89      0.73      0.80        11\n",
      "  LynnleyBrowning       0.93      1.00      0.96        13\n",
      "  MarcelMichelson       1.00      0.50      0.67        12\n",
      "     MarkBendeich       0.86      0.55      0.67        11\n",
      "       MartinWolk       0.57      0.80      0.67         5\n",
      "     MatthewBunce       1.00      0.86      0.92        14\n",
      "    MichaelConnor       0.83      0.77      0.80        13\n",
      "       MureDickie       0.44      0.40      0.42        10\n",
      "        NickLouth       0.83      1.00      0.91        10\n",
      "  PatriciaCommins       0.80      0.89      0.84         9\n",
      "    PeterHumphrey       0.35      0.89      0.50         9\n",
      "       PierreTran       0.56      0.83      0.67         6\n",
      "       RobinSidel       1.00      1.00      1.00        12\n",
      "     RogerFillion       1.00      0.88      0.93         8\n",
      "      SamuelPerry       0.78      0.50      0.61        14\n",
      "     SarahDavison       1.00      0.29      0.44        14\n",
      "      ScottHillis       0.44      0.44      0.44         9\n",
      "      SimonCowell       0.91      1.00      0.95        10\n",
      "         TanEeLyn       1.00      0.57      0.73         7\n",
      "   TheresePoletti       0.80      0.73      0.76        11\n",
      "       TimFarrand       1.00      0.77      0.87        13\n",
      "       ToddNissen       0.60      1.00      0.75         9\n",
      "     WilliamKazer       0.00      0.00      0.00        10\n",
      "\n",
      "         accuracy                           0.72       500\n",
      "        macro avg       0.75      0.74      0.71       500\n",
      "     weighted avg       0.77      0.72      0.71       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_and_evaluate(clf, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['naive_bayes_results.pkl']"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(clf, 'naive_bayes_results.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict out of sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_y, example_X = y_train[33], X_train[33]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual author: AaronPressman\n",
      "Predicted author: AaronPressman\n"
     ]
    }
   ],
   "source": [
    "print('Actual author:', example_y)\n",
    "print('Predicted author:', clf.predict([example_X])[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span>Support Vector Machines (SVM)</span><a id='trad_ml_supervised_svm'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_svm = Pipeline([\n",
    "    ('vect', TfidfVectorizer(strip_accents='unicode',\n",
    "                             lowercase = True,\n",
    "                            max_features = 1500,\n",
    "                            stop_words='english'\n",
    "                            )),\n",
    "        \n",
    "    ('clf', SVC(kernel='rbf' ,\n",
    "                C=10, gamma=0.3)\n",
    "    ),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* The SVC estimator is very sensitive to the hyperparameters!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train and show evaluation stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training set:\n",
      "0.9975\n",
      "Accuracy on testing set:\n",
      "0.83\n",
      "Classification Report:\n",
      "                   precision    recall  f1-score   support\n",
      "\n",
      "    AaronPressman       0.89      0.89      0.89         9\n",
      "       AlanCrosby       0.79      0.92      0.85        12\n",
      "   AlexanderSmith       1.00      0.70      0.82        10\n",
      "  BenjaminKangLim       0.67      0.36      0.47        11\n",
      "    BernardHickey       1.00      0.50      0.67        10\n",
      "      BradDorfman       0.78      0.88      0.82         8\n",
      " DarrenSchuettler       0.89      0.89      0.89         9\n",
      "      DavidLawder       1.00      0.60      0.75        10\n",
      "    EdnaFernandes       0.73      0.89      0.80         9\n",
      "      EricAuchard       0.73      0.89      0.80         9\n",
      "   FumikoFujisaki       1.00      1.00      1.00        10\n",
      "   GrahamEarnshaw       0.83      1.00      0.91        10\n",
      " HeatherScoffield       0.80      0.89      0.84         9\n",
      "       JanLopatka       0.67      0.44      0.53         9\n",
      "    JaneMacartney       0.36      0.50      0.42        10\n",
      "     JimGilchrist       0.88      0.88      0.88         8\n",
      "   JoWinterbottom       1.00      0.90      0.95        10\n",
      "         JoeOrtiz       0.82      1.00      0.90         9\n",
      "     JohnMastrini       0.83      0.88      0.86        17\n",
      "     JonathanBirt       0.80      1.00      0.89         8\n",
      "      KarlPenhaul       0.93      1.00      0.96        13\n",
      "        KeithWeir       0.83      1.00      0.91        10\n",
      "   KevinDrawbaugh       0.82      0.90      0.86        10\n",
      "    KevinMorrison       0.50      1.00      0.67         3\n",
      "    KirstinRidley       1.00      0.56      0.71         9\n",
      "KouroshKarimkhany       0.88      0.88      0.88         8\n",
      "        LydiaZajc       1.00      1.00      1.00        10\n",
      "   LynneO'Donnell       0.82      0.82      0.82        11\n",
      "  LynnleyBrowning       1.00      1.00      1.00        13\n",
      "  MarcelMichelson       1.00      0.67      0.80        12\n",
      "     MarkBendeich       0.79      1.00      0.88        11\n",
      "       MartinWolk       0.83      1.00      0.91         5\n",
      "     MatthewBunce       1.00      0.86      0.92        14\n",
      "    MichaelConnor       1.00      0.85      0.92        13\n",
      "       MureDickie       0.60      0.60      0.60        10\n",
      "        NickLouth       0.90      0.90      0.90        10\n",
      "  PatriciaCommins       1.00      1.00      1.00         9\n",
      "    PeterHumphrey       0.57      0.89      0.70         9\n",
      "       PierreTran       0.60      1.00      0.75         6\n",
      "       RobinSidel       1.00      1.00      1.00        12\n",
      "     RogerFillion       1.00      1.00      1.00         8\n",
      "      SamuelPerry       0.81      0.93      0.87        14\n",
      "     SarahDavison       1.00      0.71      0.83        14\n",
      "      ScottHillis       0.67      0.44      0.53         9\n",
      "      SimonCowell       1.00      0.90      0.95        10\n",
      "         TanEeLyn       0.83      0.71      0.77         7\n",
      "   TheresePoletti       0.91      0.91      0.91        11\n",
      "       TimFarrand       0.92      0.85      0.88        13\n",
      "       ToddNissen       0.90      1.00      0.95         9\n",
      "     WilliamKazer       0.27      0.30      0.29        10\n",
      "\n",
      "         accuracy                           0.83       500\n",
      "        macro avg       0.84      0.83      0.82       500\n",
      "     weighted avg       0.85      0.83      0.83       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_and_evaluate(clf_svm, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['svm_results.pkl']"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(clf_svm, 'svm_results.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict out of sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_y, example_X = y_train[33], X_train[33]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual author: AaronPressman\n",
      "Predicted author: AaronPressman\n"
     ]
    }
   ],
   "source": [
    "print('Actual author:', example_y)\n",
    "print('Predicted author:', clf_svm.predict([example_X])[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span>Model Selection and Evaluation</span><a id='trad_ml_eval'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the `TfidfVectorizer` and `SVC()` estimator take a lot of hyperparameters.  \n",
    "\n",
    "It can be difficult to figure out what the best parameters are.\n",
    "\n",
    "We can use `GridSearchCV` to help figure this out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define the options that should be tried out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_search = Pipeline([\n",
    "    ('vect', TfidfVectorizer()),\n",
    "    ('clf', SVC())\n",
    "])\n",
    "parameters = { 'vect__stop_words': ['english'],\n",
    "                'vect__strip_accents': ['unicode'],\n",
    "              'vect__max_features' : [1500],\n",
    "              'vect__ngram_range': [(1,1), (2,2) ],\n",
    "             'clf__gamma' : [0.2, 0.3, 0.4], \n",
    "             'clf__C' : [8, 10, 12],\n",
    "              'clf__kernel' : ['rbf']\n",
    "             }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run everything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=Pipeline(steps=[('vect', TfidfVectorizer()),\n",
       "                                       ('clf', SVC())]),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'clf__C': [8, 10, 12], 'clf__gamma': [0.2, 0.3, 0.4],\n",
       "                         'clf__kernel': ['rbf'], 'vect__max_features': [1500],\n",
       "                         'vect__ngram_range': [(1, 1), (2, 2)],\n",
       "                         'vect__stop_words': ['english'],\n",
       "                         'vect__strip_accents': ['unicode']},\n",
       "             scoring=make_scorer(f1_score, average=micro))"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid = GridSearchCV(clf_search, \n",
    "                    param_grid=parameters, \n",
    "                    scoring=make_scorer(f1_score, average='micro'), \n",
    "                    n_jobs=-1\n",
    "                   )\n",
    "grid.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* if you are on a powerful (preferably unix system) you can set n_jobs to the number of available threads to speed up the calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'clf__C': 8, 'clf__gamma': 0.4, 'clf__kernel': 'rbf', 'vect__max_features': 1500, 'vect__ngram_range': (1, 1), 'vect__stop_words': 'english', 'vect__strip_accents': 'unicode'} with a score of 0.79\n",
      "                   precision    recall  f1-score   support\n",
      "\n",
      "    AaronPressman       0.89      0.89      0.89         9\n",
      "       AlanCrosby       0.79      0.92      0.85        12\n",
      "   AlexanderSmith       1.00      0.70      0.82        10\n",
      "  BenjaminKangLim       0.67      0.36      0.47        11\n",
      "    BernardHickey       1.00      0.50      0.67        10\n",
      "      BradDorfman       0.78      0.88      0.82         8\n",
      " DarrenSchuettler       1.00      0.89      0.94         9\n",
      "      DavidLawder       1.00      0.60      0.75        10\n",
      "    EdnaFernandes       0.73      0.89      0.80         9\n",
      "      EricAuchard       0.73      0.89      0.80         9\n",
      "   FumikoFujisaki       1.00      1.00      1.00        10\n",
      "   GrahamEarnshaw       0.83      1.00      0.91        10\n",
      " HeatherScoffield       0.82      1.00      0.90         9\n",
      "       JanLopatka       0.60      0.33      0.43         9\n",
      "    JaneMacartney       0.36      0.50      0.42        10\n",
      "     JimGilchrist       0.88      0.88      0.88         8\n",
      "   JoWinterbottom       1.00      0.90      0.95        10\n",
      "         JoeOrtiz       0.82      1.00      0.90         9\n",
      "     JohnMastrini       0.79      0.88      0.83        17\n",
      "     JonathanBirt       0.80      1.00      0.89         8\n",
      "      KarlPenhaul       0.93      1.00      0.96        13\n",
      "        KeithWeir       0.83      1.00      0.91        10\n",
      "   KevinDrawbaugh       0.82      0.90      0.86        10\n",
      "    KevinMorrison       0.60      1.00      0.75         3\n",
      "    KirstinRidley       1.00      0.56      0.71         9\n",
      "KouroshKarimkhany       0.88      0.88      0.88         8\n",
      "        LydiaZajc       1.00      1.00      1.00        10\n",
      "   LynneO'Donnell       0.82      0.82      0.82        11\n",
      "  LynnleyBrowning       1.00      1.00      1.00        13\n",
      "  MarcelMichelson       1.00      0.67      0.80        12\n",
      "     MarkBendeich       0.73      1.00      0.85        11\n",
      "       MartinWolk       0.83      1.00      0.91         5\n",
      "     MatthewBunce       1.00      0.86      0.92        14\n",
      "    MichaelConnor       1.00      0.85      0.92        13\n",
      "       MureDickie       0.60      0.60      0.60        10\n",
      "        NickLouth       0.90      0.90      0.90        10\n",
      "  PatriciaCommins       1.00      1.00      1.00         9\n",
      "    PeterHumphrey       0.57      0.89      0.70         9\n",
      "       PierreTran       0.60      1.00      0.75         6\n",
      "       RobinSidel       1.00      1.00      1.00        12\n",
      "     RogerFillion       1.00      1.00      1.00         8\n",
      "      SamuelPerry       0.76      0.93      0.84        14\n",
      "     SarahDavison       1.00      0.71      0.83        14\n",
      "      ScottHillis       0.67      0.44      0.53         9\n",
      "      SimonCowell       1.00      0.90      0.95        10\n",
      "         TanEeLyn       0.83      0.71      0.77         7\n",
      "   TheresePoletti       0.90      0.82      0.86        11\n",
      "       TimFarrand       0.92      0.85      0.88        13\n",
      "       ToddNissen       0.90      1.00      0.95         9\n",
      "     WilliamKazer       0.27      0.30      0.29        10\n",
      "\n",
      "         accuracy                           0.83       500\n",
      "        macro avg       0.84      0.83      0.82       500\n",
      "     weighted avg       0.85      0.83      0.82       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"The best parameters are %s with a score of %0.2f\" % (grid.best_params_, grid.best_score_))\n",
    "y_true, y_pred = y_test, grid.predict(X_test)\n",
    "print(metrics.classification_report(y_true, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span>Unsupervised</span><a id='trad_ml_unsupervised'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span>Latent Dirichilet Allocation (LDA)</span><a id='trad_ml_unsupervised_lda'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectorizer (using countvectorizer for the sake of example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(strip_accents='unicode',\n",
    "                             lowercase = True,\n",
    "                            max_features = 1500,\n",
    "                            stop_words='english', max_df=0.8)\n",
    "tf_large = vectorizer.fit_transform(clean_paragraphs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the LDA model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_topics = 10\n",
    "n_top_words = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda = LatentDirichletAllocation(n_components=n_topics, max_iter=10,\n",
    "                                learning_method='online',\n",
    "                                n_jobs=-1)\n",
    "lda_fitted = lda.fit_transform(tf_large)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize top words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_top_words(model, feature_names, n_top_words):\n",
    "    out_list = []\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        out_list.append((topic_idx+1, \" \".join([feature_names[i] for i in topic.argsort()[:-n_top_words - 1:-1]])))\n",
    "    out_df = pd.DataFrame(out_list, columns=['topic_id', 'top_words'])\n",
    "    return out_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_df = save_top_words(lda, vectorizer.get_feature_names(), n_top_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_id</th>\n",
       "      <th>top_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>company pound million share group business bil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>official south people north northern police ko...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>china kong hong beijing chinese deng year part...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>market percent price year china trade bank sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>percent year million profit analyst quarter sh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>gold bre canada toronto stock busang canadian ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>ford union oil strike car plant new worker pro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>tonne year million crop new world export add a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>company service new network computer internet ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>bank financial fund nomura company loan firm s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   topic_id                                          top_words\n",
       "0         1  company pound million share group business bil...\n",
       "1         2  official south people north northern police ko...\n",
       "2         3  china kong hong beijing chinese deng year part...\n",
       "3         4  market percent price year china trade bank sta...\n",
       "4         5  percent year million profit analyst quarter sh...\n",
       "5         6  gold bre canada toronto stock busang canadian ...\n",
       "6         7  ford union oil strike car plant new worker pro...\n",
       "7         8  tonne year million crop new world export add a...\n",
       "8         9  company service new network computer internet ...\n",
       "9        10  bank financial fund nomura company loan firm s..."
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span>pyLDAvis</span><a id='trad_ml_unsupervised_pyLDAvis'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pyLDAvis\n",
    "import pyLDAvis.sklearn\n",
    "pyLDAvis.enable_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el49562867804621896767790924\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el49562867804621896767790924_data = {\"mdsDat\": {\"x\": [-0.06940003653598698, -0.19408117995720842, -0.037245380722667525, 0.18546081077318854, -0.0005391457036158938, -0.07526202175593534, 0.016380758173715457, 0.03096064384608569, 0.2444652611618537, -0.10073970927942939], \"y\": [0.007189234320693191, -0.07683297628721177, -0.029790341736024815, -0.01564337637894662, -0.006789133907126656, -0.06738577633113985, -0.09313497572731662, -0.014621801195176323, 0.04114212492228232, 0.2558670223199672], \"topics\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], \"cluster\": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"Freq\": [17.32891001920437, 16.906264950483223, 14.55654063454666, 13.805324329934027, 8.437633963587201, 7.977310100416945, 7.18718004557624, 6.40734977126214, 4.057657253838257, 3.335828931150938]}, \"tinfo\": {\"Term\": [\"bank\", \"china\", \"kong\", \"hong\", \"percent\", \"pound\", \"stock\", \"tonne\", \"million\", \"market\", \"company\", \"share\", \"beijing\", \"quarter\", \"profit\", \"gold\", \"chinese\", \"bre\", \"price\", \"canada\", \"official\", \"year\", \"people\", \"financial\", \"trade\", \"deng\", \"sale\", \"billion\", \"report\", \"earnings\", \"itt\", \"cigarette\", \"philip\", \"hilton\", \"stg\", \"suez\", \"pounds\", \"wmx\", \"flotation\", \"pound\", \"morris\", \"tobacco\", \"mercury\", \"bonus\", \"franchise\", \"penny\", \"uk\", \"florida\", \"shareholder\", \"england\", \"waste\", \"consolidation\", \"sainsbury\", \"plc\", \"acquisition\", \"chemical\", \"royal\", \"shares\", \"liability\", \"london\", \"club\", \"board\", \"britain\", \"british\", \"merger\", \"stake\", \"company\", \"group\", \"executive\", \"share\", \"buy\", \"offer\", \"value\", \"deal\", \"million\", \"management\", \"business\", \"cash\", \"chief\", \"asset\", \"billion\", \"amp\", \"investment\", \"market\", \"stock\", \"analyst\", \"percent\", \"year\", \"industry\", \"new\", \"plan\", \"add\", \"mainframe\", \"cereal\", \"kellogg\", \"surcharge\", \"sterling\", \"cba\", \"quarter\", \"sales\", \"fiscal\", \"sears\", \"earnings\", \"item\", \"expectation\", \"margin\", \"income\", \"jones\", \"consensus\", \"quaker\", \"outlook\", \"reduction\", \"profit\", \"net\", \"snapple\", \"expense\", \"tax\", \"earn\", \"revenue\", \"profitability\", \"forecast\", \"half\", \"growth\", \"result\", \"cent\", \"sale\", \"percent\", \"increase\", \"rise\", \"million\", \"year\", \"analyst\", \"ibm\", \"cut\", \"performance\", \"compare\", \"share\", \"expect\", \"fall\", \"billion\", \"report\", \"strong\", \"cost\", \"rate\", \"product\", \"market\", \"price\", \"business\", \"company\", \"stock\", \"end\", \"new\", \"online\", \"aol\", \"mci\", \"csf\", \"aircraft\", \"plane\", \"satellite\", \"airbus\", \"wireless\", \"lagardere\", \"aerospatiale\", \"internet\", \"thomson\", \"sprint\", \"communications\", \"nynex\", \"dassault\", \"telecommunications\", \"distance\", \"microsoft\", \"cable\", \"windows\", \"digital\", \"modem\", \"cellular\", \"alcatel\", \"fcc\", \"web\", \"aerospace\", \"subscriber\", \"telecom\", \"tv\", \"phone\", \"network\", \"apple\", \"boeing\", \"software\", \"computer\", \"technology\", \"service\", \"electronic\", \"user\", \"telecommunication\", \"customer\", \"french\", \"france\", \"company\", \"new\", \"access\", \"television\", \"plan\", \"industry\", \"market\", \"offer\", \"long\", \"corp\", \"year\", \"group\", \"billion\", \"government\", \"state\", \"business\", \"tung\", \"handover\", \"jiang\", \"democracy\", \"legislature\", \"democratic\", \"colony\", \"jail\", \"dissident\", \"provisional\", \"patten\", \"zemin\", \"chee\", \"hwa\", \"activist\", \"liberty\", \"sovereignty\", \"mao\", \"tsang\", \"corruption\", \"prison\", \"sino\", \"declaration\", \"sentence\", \"human\", \"deng\", \"xinhua\", \"freedom\", \"lee\", \"territory\", \"taiwan\", \"kong\", \"hong\", \"political\", \"party\", \"china\", \"beijing\", \"visit\", \"chinese\", \"communist\", \"leader\", \"right\", \"law\", \"civil\", \"xiaoping\", \"diplomat\", \"people\", \"rule\", \"death\", \"july\", \"official\", \"government\", \"british\", \"year\", \"economic\", \"state\", \"president\", \"tell\", \"foreign\", \"issue\", \"plan\", \"nomura\", \"encryption\", \"banking\", \"mgam\", \"underwrite\", \"scandal\", \"loan\", \"yen\", \"probe\", \"investigation\", \"mutual\", \"bank\", \"trust\", \"pension\", \"watchdog\", \"insurance\", \"scheme\", \"regulation\", \"branch\", \"australian\", \"melbourne\", \"amro\", \"australia\", \"institution\", \"regulate\", \"regulator\", \"abn\", \"client\", \"security\", \"bond\", \"fund\", \"credit\", \"financial\", \"japan\", \"brokerage\", \"securities\", \"payment\", \"firm\", \"life\", \"problem\", \"finance\", \"allow\", \"business\", \"commercial\", \"investment\", \"government\", \"company\", \"policy\", \"market\", \"big\", \"issue\", \"year\", \"tell\", \"management\", \"industry\", \"official\", \"report\", \"cnb\", \"crown\", \"yuan\", \"skoda\", \"index\", \"inflation\", \"trader\", \"economist\", \"sugar\", \"shanghai\", \"deficit\", \"klaus\", \"monetary\", \"casino\", \"currency\", \"enterprise\", \"central\", \"shenzhen\", \"fix\", \"surplus\", \"steel\", \"stable\", \"prague\", \"economy\", \"domestic\", \"import\", \"lme\", \"factor\", \"point\", \"czech\", \"investor\", \"price\", \"market\", \"rate\", \"trade\", \"foreign\", \"state\", \"percent\", \"china\", \"debt\", \"high\", \"stock\", \"rise\", \"year\", \"bank\", \"billion\", \"analyst\", \"chinese\", \"government\", \"week\", \"share\", \"expect\", \"crop\", \"coast\", \"cocoa\", \"cargo\", \"ivory\", \"tonne\", \"apec\", \"bean\", \"corn\", \"kmart\", \"harvest\", \"cathay\", \"grain\", \"port\", \"rain\", \"freight\", \"farmer\", \"coffee\", \"airport\", \"flight\", \"arrival\", \"season\", \"airline\", \"factory\", \"singapore\", \"exporter\", \"grade\", \"air\", \"film\", \"shipment\", \"passenger\", \"asia\", \"quality\", \"capacity\", \"export\", \"world\", \"year\", \"source\", \"mid\", \"april\", \"million\", \"add\", \"new\", \"area\", \"total\", \"open\", \"reuters\", \"industry\", \"country\", \"tell\", \"market\", \"kong\", \"hong\", \"ford\", \"russia\", \"russian\", \"uaw\", \"textile\", \"norfolk\", \"gazprom\", \"moscow\", \"chrysler\", \"automaker\", \"wage\", \"johnson\", \"truck\", \"car\", \"worker\", \"norilsk\", \"auto\", \"strike\", \"oil\", \"union\", \"workers\", \"soviet\", \"detroit\", \"plant\", \"csx\", \"labor\", \"vehicle\", \"gm\", \"seat\", \"pact\", \"gas\", \"conrail\", \"production\", \"contract\", \"czech\", \"agreement\", \"talk\", \"job\", \"new\", \"export\", \"trade\", \"company\", \"year\", \"plan\", \"percent\", \"day\", \"industry\", \"hwang\", \"xinjiang\", \"ce\", \"pyongyang\", \"korean\", \"bzw\", \"moslem\", \"riot\", \"guerrilla\", \"seoul\", \"packer\", \"bomb\", \"embassy\", \"kill\", \"electric\", \"northern\", \"natwest\", \"electricity\", \"ethnic\", \"arrest\", \"panel\", \"korea\", \"rebel\", \"police\", \"bus\", \"town\", \"colombia\", \"south\", \"army\", \"appeal\", \"north\", \"official\", \"bid\", \"people\", \"fight\", \"beijing\", \"region\", \"chinese\", \"government\", \"capital\", \"leader\", \"state\", \"southern\", \"foreign\", \"source\", \"bre\", \"gold\", \"busang\", \"copper\", \"toronto\", \"freeport\", \"indonesian\", \"nickel\", \"lloyd\", \"barrick\", \"ounce\", \"walsh\", \"strathcona\", \"minerals\", \"eaton\", \"names\", \"canadian\", \"hughes\", \"metal\", \"sample\", \"zinc\", \"canada\", \"indonesia\", \"discovery\", \"test\", \"lme\", \"deposit\", \"consultant\", \"contain\", \"rich\", \"project\", \"stock\", \"exchange\", \"report\", \"base\", \"company\", \"trade\", \"million\", \"share\", \"billion\", \"president\", \"percent\", \"government\"], \"Freq\": [3158.0, 3599.0, 2394.0, 2379.0, 5501.0, 1576.0, 2111.0, 970.0, 5023.0, 4486.0, 5479.0, 3333.0, 1410.0, 1498.0, 1865.0, 612.0, 1642.0, 575.0, 2168.0, 515.0, 1656.0, 6518.0, 1496.0, 1299.0, 1830.0, 789.0, 2012.0, 3254.0, 1594.0, 938.0, 205.44764817741603, 197.95156685547695, 151.55709781194327, 128.2900336875436, 199.27101450302928, 124.82405835252979, 123.21003660878638, 95.67619378330863, 84.25856157808312, 1558.5327189507525, 128.10306798395254, 356.81930632017753, 152.14036117606966, 85.72322796065149, 121.06709561310326, 475.99446446125944, 184.3369988978126, 94.43687755487957, 748.945517072113, 83.77863313505638, 88.99860434229083, 94.8465257954407, 64.5324265067759, 443.2217019841648, 481.4867898083673, 106.55389494922764, 93.87688430092741, 105.18416030836603, 72.19951627500198, 406.7900936954303, 125.36227330919291, 302.4681211383282, 467.58179722655694, 755.3095816379998, 450.67118117310065, 371.2449486260204, 2241.243862651815, 1029.2938265031787, 657.9236556152906, 1355.5835011738652, 522.3132015842335, 618.3367921662834, 401.5182008164161, 636.8194542590991, 1502.4327496260316, 344.3039770666336, 830.4886794140543, 339.85486079540345, 480.9525690990059, 315.1102793278904, 812.5519680546082, 301.42284198485413, 400.51032591990776, 732.8410705795364, 525.0929501599219, 590.7563244908025, 648.9538857572539, 593.0308755105444, 428.00557405320967, 485.99236308055674, 439.24456688217464, 414.9041988963321, 146.94758795358103, 143.77283012759443, 128.19561691908515, 72.0900412951579, 82.14565254852539, 62.42617201700649, 1458.1795535779386, 122.58066796710216, 223.74804221778092, 78.71183772838667, 862.3498502497432, 104.08065735361376, 211.75564825768384, 348.926512658195, 264.2869601831791, 81.03714679208429, 82.43578357484571, 99.3458595992964, 102.83475757603661, 83.32401767024848, 1586.4191248475631, 466.573500356241, 93.13788788059422, 89.56448347506364, 412.4654158752234, 165.0858104794818, 723.4968291541084, 66.09794574959412, 424.0407861284634, 626.4859829341397, 822.0226127727828, 776.7656569491655, 468.2594013147198, 1332.3583984344616, 3008.656883736818, 703.8663956977343, 966.810778588327, 2451.1493335560517, 2831.740420574855, 1548.6862987815152, 286.6515606671496, 560.4631082148899, 229.32264793894814, 305.7371336566126, 1393.7514585329857, 1075.2557549754579, 583.4490827019462, 1265.4374252294563, 711.7263316569449, 472.0869985688463, 593.8212408099427, 579.6459237796931, 546.1822488249024, 1054.790483807253, 719.3559749687839, 746.7387124884309, 816.4314263277589, 570.9940643430884, 505.58947208408085, 532.0555182212547, 326.4417808587422, 248.26916471864283, 260.26986980331174, 305.48251646952184, 198.29781604396294, 168.54991729546703, 230.69261045801602, 155.46064810229328, 232.70249007975104, 162.78378816114406, 143.65244008114772, 686.2778462836471, 512.6160126331423, 95.71285381519206, 328.86855529270997, 90.29231484787479, 82.97504547226015, 89.63830542934312, 285.2744321250951, 467.71153189060266, 340.5401148478977, 154.12476106936444, 287.2867037779069, 76.05787850488971, 85.44406447727876, 163.6577552401139, 208.15640116809536, 139.5622164811665, 85.07712850856161, 160.76558618502887, 356.03066825914544, 253.9150631982423, 298.6186978430534, 722.4297867123901, 407.62854402808176, 300.194217064632, 502.4975406010773, 691.1483125234506, 407.2977733972864, 1071.5060807198347, 320.78761661034616, 219.4101794038388, 209.9448561206289, 483.54803644342127, 320.9113593513536, 270.57526674059716, 1405.7162369246623, 1005.3150228555818, 304.5563038647436, 313.37122520583347, 610.8334181160658, 487.00621311425664, 664.1586200449183, 405.66287636301615, 407.68340263560043, 328.46304209431304, 438.32589311835864, 378.31630929112146, 362.9532686313235, 333.4239840212908, 324.5406792890588, 323.9107634464754, 391.0273685000363, 298.6575892036122, 270.9680666311759, 212.06681903351262, 182.61535016503385, 174.31961527587387, 146.56698496348568, 122.67920944022501, 113.60793824493359, 112.88061409447857, 111.91053184830925, 102.96137386774974, 93.69862376156082, 93.69850558743398, 91.15384424689967, 105.7585792868042, 85.24696870421572, 86.35248255574392, 82.26781362814086, 84.55582077236373, 73.89866111101304, 165.02498423028175, 68.93387081402454, 65.91222301618667, 320.3371320866392, 762.2130484307186, 89.18729744315026, 157.1781002195889, 170.1030166312378, 340.35789635764996, 380.0390492651938, 2198.3308701773294, 2183.851251405621, 557.5798508446998, 692.8071399796781, 2670.0280964940202, 1126.8366082496896, 253.54160080883852, 1028.2722090719608, 332.4891065435232, 522.1949324469232, 568.1024331886638, 432.73625032970574, 197.6491202428292, 164.73708016387297, 168.13322792864577, 672.7000755048763, 425.12511176143545, 219.25663357409798, 266.7472705198417, 468.1288828041199, 499.6509732733547, 388.13904752538303, 699.9422283319955, 262.6296741197348, 334.19456212988086, 287.66329519364535, 323.62000159199306, 284.53753968630826, 281.93043894713276, 279.2214404270412, 389.2178182464965, 99.50302215022727, 78.26636575581507, 98.17048737131707, 92.03220983948391, 164.3498854021871, 348.6474903291677, 160.6270078826381, 107.73972095872954, 184.9009622807692, 113.37238324875707, 2564.3953790036708, 174.5692498295888, 110.55877132818509, 77.9567145493735, 313.9148218397278, 64.3771496542978, 127.02639354465462, 60.90817388185911, 229.55533920651152, 76.22525246883869, 56.79451701803951, 329.0818360816408, 103.20510190145926, 61.01487988312659, 155.32500127757103, 58.95568100239115, 147.6213453734371, 335.6806589185677, 118.55071957540633, 474.9071989382337, 229.51489761929454, 657.4016448223944, 259.56202475940444, 163.36804803925378, 240.53153054953248, 165.78707149894893, 338.20828948946615, 178.98905428699078, 268.9663452847532, 215.59642755566134, 227.37079776348764, 333.7725241000232, 166.57106617724295, 237.60219999905706, 298.51821387323315, 388.0388190545616, 186.83621713012337, 298.80963769698093, 222.68370253849568, 195.95393731898434, 246.63139392426146, 197.89077401981297, 178.3521579409424, 189.44099356794098, 184.92576279107203, 184.5149043748437, 76.85760355932608, 425.6487077913579, 357.74173260556665, 92.37053344010559, 270.2121872714963, 276.7567753117202, 455.3354222206588, 178.3647503945231, 102.72199230666324, 325.82913189116675, 193.1031038965101, 103.9459123137026, 72.73913008827758, 102.87914687863696, 185.79134263146426, 98.60789049677088, 378.85171857579536, 82.42091307149526, 107.99089646504035, 52.23980530559119, 65.9927187459013, 48.45257124254456, 134.11167604310543, 210.4410625365117, 245.5464033810909, 191.35438948155507, 58.793532717919184, 93.08660734339666, 293.8917461317626, 294.53519627650587, 309.83033982683185, 758.4133291431086, 1328.9204971865577, 412.7548067504658, 574.0287115538968, 341.6888335296667, 459.4601623982683, 1103.3924464450304, 670.3242095178015, 225.49622610185182, 374.2415471540908, 431.56329957233436, 367.07061590647834, 711.8415926434492, 474.06566135822277, 404.33384443470834, 396.0956023699289, 295.06208022445577, 288.4951376852289, 268.1663066003145, 305.7749782763546, 275.7511420132241, 341.5416621272872, 247.81547169486564, 220.98078669142177, 226.50890501057296, 183.4146796105304, 963.0695385240732, 109.13537870011704, 117.83911685623902, 163.620442284427, 89.423977902925, 111.1412815407611, 86.27735355668719, 132.57595471650842, 167.9240868767389, 102.38557970374833, 69.6661534657342, 86.16541194577876, 181.72796446532797, 147.29338743932843, 157.29364513795343, 86.99153223247134, 107.50251035686394, 191.4884623296165, 128.77206654665562, 133.01612354612442, 99.78325510822286, 71.7470741260261, 250.80766255791465, 104.70771145380472, 68.02714777411889, 108.69550246188092, 193.97136313419574, 174.6997853706932, 162.36007550837394, 281.98678506787667, 300.3041518811612, 669.6931532943622, 242.43316916247377, 138.40852495621, 158.60971121745297, 396.83098023610484, 264.41960109325726, 327.77293013170265, 179.76809721780012, 190.8177888738675, 181.65173013101727, 184.6972053172435, 204.72980173702126, 178.0064395592658, 190.08425141911118, 208.80561482659562, 194.38576441792307, 194.16106633818168, 486.17446318203895, 300.4067847637546, 197.55748155546578, 171.39040072405865, 191.80666156986126, 175.61196420281118, 91.09695986073771, 77.82399621555834, 226.04702335523166, 162.9353000624919, 182.4490344041251, 99.31776878045773, 257.544755161252, 373.8130936407584, 338.42789292906036, 138.08207302960136, 129.59746518318443, 407.3458624633746, 414.1529403393892, 418.337971169851, 80.39654124935817, 68.71535531531003, 117.19180513165084, 371.5052148661234, 159.7041985490783, 81.01746892907272, 226.05205117180228, 267.052062536422, 173.45132117692765, 92.92509622157232, 179.8783651387561, 223.04976732028558, 331.1870974587875, 278.9108351227477, 319.46063298105935, 228.07421342456746, 265.5160450943296, 195.6282042300292, 348.95753492655354, 211.88542551307884, 253.06665260430094, 266.21255527497834, 243.40987389937905, 207.64812197983724, 214.23911369899994, 188.46786667456027, 189.72613633880198, 189.07384010559255, 178.31929156690737, 141.45702822104005, 122.90383940112551, 241.58562072673305, 96.72309047455227, 90.92570654923378, 114.30040846112077, 101.11745231534313, 156.990452299566, 68.4375895591858, 108.77037994302266, 89.51341188879879, 165.22549061115458, 223.74633103711727, 277.26859189866127, 172.75710614655728, 146.89222713809818, 103.48730454908909, 75.48597241656573, 92.1867234311831, 199.1459522870633, 80.7985288447371, 260.95409800918867, 96.06525824061326, 82.48703962985755, 90.49457658194068, 316.15890709596266, 126.29853160435249, 79.47863726551485, 293.60619065661194, 362.12426258234507, 238.59775495187193, 296.87395676307483, 116.19204023246007, 221.196167997055, 137.13511917224903, 212.99538977545242, 184.93083875548163, 134.76053533255637, 117.8238705553829, 115.75889918259021, 106.82673569085708, 110.19593062551962, 109.98948320828485, 574.6978475655216, 611.6596767623647, 331.41499286482957, 267.654004371387, 370.3802990905799, 169.63630571787266, 155.49817836311874, 140.14672130017337, 219.85333086252436, 123.00814145518993, 100.55256040093435, 98.72410301614536, 96.23026122113474, 81.97926451127265, 82.3004308232094, 101.66174922714896, 323.100815783191, 95.93581167857783, 266.47794392517346, 74.94552502880643, 135.29014863778488, 437.65434626798566, 114.71064823163026, 49.84175030098499, 93.7779854259482, 52.91297724642251, 119.8908278117101, 38.8747829784587, 29.67835372838836, 51.139754040948304, 146.24196275032668, 365.5314215690461, 144.9490361386129, 198.1582227065132, 154.0872614539729, 222.69363880792437, 154.26144330821467, 185.59026395352947, 157.69729423257718, 130.38717214287936, 107.5127217166282, 109.59178272015858, 105.2673640153835], \"Total\": [3158.0, 3599.0, 2394.0, 2379.0, 5501.0, 1576.0, 2111.0, 970.0, 5023.0, 4486.0, 5479.0, 3333.0, 1410.0, 1498.0, 1865.0, 612.0, 1642.0, 575.0, 2168.0, 515.0, 1656.0, 6518.0, 1496.0, 1299.0, 1830.0, 789.0, 2012.0, 3254.0, 1594.0, 938.0, 206.36573691201065, 199.16734017367114, 152.49168759794924, 129.2035548532585, 200.72281433666927, 125.78724632484914, 124.22190055626822, 96.61119091274075, 85.16916159845731, 1576.9093700456401, 130.31206501201223, 363.4554135438614, 156.80076462852023, 88.36418654612831, 127.10828392372672, 500.8665578968616, 194.45507987204402, 101.06808270667098, 854.6501005443541, 97.04281173476465, 105.80256218115198, 113.08644995042987, 77.4483044483928, 532.9825226758911, 581.6397968933567, 130.2324981121207, 119.16779415886437, 135.5754667188974, 94.31326196882007, 533.7696880290151, 169.65241339411472, 444.8638907416487, 728.7189986277156, 1263.5866212082099, 728.7260092194862, 595.9684711013066, 5479.738585485954, 2191.5137078873786, 1259.0750709931353, 3333.8063707992133, 971.7515175517418, 1233.4662628066988, 689.8505368944998, 1410.811498834872, 5023.136851717023, 602.0140374996771, 2396.691150801944, 609.2099849619186, 1096.5698107671074, 549.1928311714286, 3254.0948785375103, 512.7884151571621, 1049.529459828699, 4486.536893860493, 2111.1124554555695, 3162.275339796167, 5501.563943183113, 6518.509471309362, 1742.267595351884, 3343.193819615464, 2105.2267721084145, 1656.0604938815661, 147.872806320033, 144.7816709706702, 129.1597653831579, 73.01306610460468, 83.43623458417152, 63.47231364765841, 1498.484155999643, 126.03910999078252, 232.20319447682283, 81.93514078169683, 938.2098861402715, 115.16461153593701, 237.07038814527283, 393.04358656452723, 298.0579691942466, 91.98626645468688, 93.8313858520987, 113.2315802013323, 118.1094890017725, 97.26295070924049, 1865.5091252013872, 556.1077263548572, 111.14700742828278, 108.96953855766242, 503.29495523055635, 202.85913616439638, 892.5343340978516, 82.06729104149231, 527.9324786072212, 788.6240078065, 1073.6695616387578, 1093.6078828880134, 637.983764728483, 2012.9253045791404, 5501.563943183113, 1058.702291046458, 1554.0802325306918, 5023.136851717023, 6518.509471309362, 3162.275339796167, 399.38139355486555, 918.6738460769786, 308.4385835799158, 450.3353645404089, 3333.8063707992133, 2467.755920038936, 1084.2087069209795, 3254.0948785375103, 1594.2174729228327, 860.7636794067184, 1231.7208360153238, 1186.539932042762, 1180.7410561788927, 4486.536893860493, 2168.5712806790234, 2396.691150801944, 5479.738585485954, 2111.1124554555695, 1478.7669426334296, 3343.193819615464, 327.3671233774796, 249.18587270169255, 261.28201030242707, 306.8075480140402, 199.21535804119276, 169.46831819562934, 231.99185011591922, 156.384042810387, 234.13772259223632, 163.85426444766125, 144.6268388413371, 691.2587507067934, 516.7714900621726, 96.62880244280359, 332.0293679744628, 91.23596562495486, 83.90242423283436, 90.70508147639333, 289.18217833885353, 474.68496167359757, 346.10029773809555, 156.8441613235794, 293.0249875795837, 77.96928408276729, 88.05640180415561, 168.70276534259912, 215.49064792974136, 144.66675122658333, 88.41806401373444, 167.41853245373767, 377.963657237102, 267.725311191521, 318.78692146229224, 793.0013616006128, 442.70226354622343, 321.5075027430965, 564.4138393537702, 848.2534090228017, 480.10842994279653, 1510.553395051293, 381.34480974236493, 250.3850380288323, 239.28953338454485, 824.3208751446044, 448.1914880594784, 358.69782276294256, 5479.738585485954, 3343.193819615464, 452.9868883517965, 486.042608991078, 2105.2267721084145, 1742.267595351884, 4486.536893860493, 1233.4662628066988, 1271.663970839491, 687.7966581928847, 6518.509471309362, 2191.5137078873786, 3254.0948785375103, 1978.7374376230136, 1503.5321793738406, 2396.691150801944, 391.95190814942913, 299.58583827654724, 272.1360310379968, 212.99189535753885, 183.53972844248736, 175.26036683657298, 147.501453240359, 123.60293549001902, 114.5395079741616, 113.80628484114241, 112.83480074604292, 103.91458878745557, 94.6216232458507, 94.62165072326295, 92.08347211993403, 106.85522676874629, 86.1796012603849, 87.31653001336896, 83.19209162256759, 85.55505970380263, 74.82178595283739, 167.29321577055663, 69.89669789056532, 66.83587507042601, 328.99087547522606, 789.7866946090555, 94.19985312381004, 166.14594749077617, 179.82600043714964, 360.55111504850004, 405.5787292041951, 2394.517284045658, 2379.6694030473577, 621.1518040440752, 824.6260040725994, 3599.2754725439377, 1410.2807348343274, 295.0017235618647, 1642.2787206698156, 431.54854252400213, 753.5117223469139, 896.7619100975605, 657.5503749148166, 240.21407140969268, 189.38947114345402, 196.51117557959157, 1496.151306491435, 767.6443571693379, 291.19544056299395, 432.81900142293125, 1656.6800339718911, 1978.7374376230136, 1263.5866212082099, 6518.509471309362, 624.2640934255024, 1503.5321793738406, 887.9536341028347, 1611.5963109677846, 974.0916319857008, 1096.0719191084565, 2105.2267721084145, 391.67880647638987, 100.64190346925704, 79.6120516958944, 100.33686801509165, 96.3317893657236, 174.13263242150404, 383.0386796573777, 178.91378193209408, 125.03622037692067, 223.36270603486545, 137.52082015270884, 3158.6601817983674, 221.54808811891795, 140.5013520976473, 101.92310982700836, 426.22918527097613, 88.3514165615656, 174.93247201771584, 84.08938947017361, 318.7946747051872, 106.64852386332558, 82.47296576833973, 483.27448922796344, 151.71309306075142, 89.74958528826521, 229.47500583040193, 88.01483571877071, 233.023555339115, 535.924654482429, 191.4033553574743, 826.8826221214858, 389.83291366050946, 1299.1947425171204, 479.3725650712515, 280.55008960263035, 457.3901471657084, 318.296461535807, 1079.2486166715244, 389.35471066483734, 786.4424925151833, 564.0948292645265, 764.1311916390104, 2396.691150801944, 389.6529426631082, 1049.529459828699, 1978.7374376230136, 5479.738585485954, 592.5577813698476, 4486.536893860493, 1437.240331617063, 1096.0719191084565, 6518.509471309362, 1611.5963109677846, 602.0140374996771, 1742.267595351884, 1656.6800339718911, 1594.2174729228327, 77.77897296432461, 434.7444008124781, 367.68201141832066, 95.64201731667589, 280.38165007814246, 288.2227650870238, 485.0254021302622, 196.93654903118448, 114.42229167792925, 372.38879794356205, 226.13100355328706, 127.70438605038906, 89.82621483414513, 150.00882338544736, 271.43437509042127, 149.30045444072928, 580.2002626689648, 133.12637547397776, 178.57900920747267, 87.78948403649991, 111.88857960189142, 82.69123691679594, 233.6961735639505, 375.6322508072181, 450.4462666322337, 357.0529956624634, 113.8758337642673, 193.83366582332098, 612.6236435887236, 615.8158106417054, 736.1846281947232, 2168.5712806790234, 4486.536893860493, 1186.539932042762, 1830.7027467558771, 974.0916319857008, 1503.5321793738406, 5501.563943183113, 3599.2754725439377, 616.6701735346847, 1592.6147733461123, 2111.1124554555695, 1554.0802325306918, 6518.509471309362, 3158.6601817983674, 3254.0948785375103, 3162.275339796167, 1642.2787206698156, 1978.7374376230136, 1441.5403766413, 3333.8063707992133, 2467.755920038936, 342.5842398820481, 248.73019161327193, 221.90296437735157, 227.54604633142915, 184.71825429993098, 970.7508406164253, 110.07974871638066, 118.87901431285643, 165.29025628945652, 90.33963665977535, 112.41415415660772, 87.3573958513567, 136.49853050191015, 179.70383875179644, 109.97101150504541, 75.59438993704121, 94.61246826996374, 202.10309882631645, 164.7210584038456, 185.33582866858922, 105.44373231563515, 130.522778175638, 235.963689908385, 161.99339284685985, 167.53848374384404, 128.97254502534417, 96.69156305213785, 346.3151483839936, 146.74240382568544, 96.7490122436071, 157.20929617673386, 318.81784767403644, 292.8978053719759, 266.2550204664775, 732.0422739837062, 1149.8164856996207, 6518.509471309362, 895.9642252158826, 266.84635571040695, 394.224687462564, 5023.136851717023, 1656.0604938815661, 3343.193819615464, 588.222345986571, 781.0939828884597, 688.6769936559539, 852.9418917901758, 1742.267595351884, 868.0874668868641, 1611.5963109677846, 4486.536893860493, 2394.517284045658, 2379.6694030473577, 487.2890733492809, 301.5283587916546, 198.48929760108757, 172.31691484146677, 192.88007802252852, 176.6546980050801, 92.02514647661846, 78.74364348105506, 231.07802584058095, 168.0690681029338, 194.5439151406288, 108.08111893776632, 288.27632503174425, 418.8926765528726, 388.5557517709648, 161.06601756220445, 152.64885337202008, 487.7238118417572, 497.4120112129803, 508.5075528353755, 97.81050447918074, 84.41340048234744, 144.93165846410295, 461.66112328586206, 199.71065322177995, 102.24680548264524, 286.7237667447778, 352.79846746105244, 232.53580654176193, 126.61492048740773, 248.06494310407913, 328.0875478090242, 554.6972440294378, 452.7453760250982, 615.8158106417054, 494.01574778694817, 847.559344120595, 516.1708985463747, 3343.193819615464, 732.0422739837062, 1830.7027467558771, 5479.738585485954, 6518.509471309362, 2105.2267721084145, 5501.563943183113, 1062.3347345584455, 1742.267595351884, 189.9935881459795, 179.27944879493103, 142.37134590410153, 123.82560897062562, 243.75990870512032, 97.63761571234922, 91.88001686490895, 115.55102527471126, 103.60991622222632, 162.12499166042963, 71.5745688822609, 115.571026029787, 97.43678304505708, 181.96083271715926, 258.21990673890355, 320.4039976807428, 200.03545522173994, 178.00566043749998, 126.01768643215162, 91.93983552928341, 112.7692479167508, 247.09457241370117, 101.55664961754202, 344.55402617897033, 141.1354619650935, 122.8088423201304, 147.83190287406362, 523.0664149613692, 231.35725112197895, 147.7682358449737, 592.4315233465705, 1656.6800339718911, 845.6194545913827, 1496.151306491435, 259.1213985878402, 1410.2807348343274, 412.007401794631, 1642.2787206698156, 1978.7374376230136, 806.5618075729166, 753.5117223469139, 1503.5321793738406, 324.2206677383368, 974.0916319857008, 895.9642252158826, 575.6647664687487, 612.7190600161575, 332.33709443619085, 268.62395118182377, 371.9349419198486, 170.55379285023767, 156.42055278244447, 141.0662453962087, 221.33860798938773, 123.91161262011173, 101.47436357138483, 99.64072949452775, 97.15984694873, 82.88126564189899, 83.22389155651102, 106.66218407245663, 339.882173452013, 106.29043505044979, 297.6040755079451, 83.72146973562232, 152.34365430057807, 515.494232726509, 163.4118533504211, 77.40262572633755, 198.74853893021975, 113.8758337642673, 266.05687796560187, 101.12772813444965, 77.54805369091085, 150.85189466200106, 438.67994552861137, 2111.1124554555695, 797.1693113009319, 1594.2174729228327, 1366.9332556955098, 5479.738585485954, 1830.7027467558771, 5023.136851717023, 3333.8063707992133, 3254.0948785375103, 887.9536341028347, 5501.563943183113, 1978.7374376230136], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -6.0183, -6.0555, -6.3225, -6.4892, -6.0488, -6.5166, -6.5296, -6.7825, -6.9096, -3.992, -6.4906, -5.4663, -6.3187, -6.8924, -6.5471, -5.1781, -6.1267, -6.7955, -4.7248, -6.9153, -6.8549, -6.7912, -7.1763, -5.2494, -5.1666, -6.6748, -6.8015, -6.6878, -7.064, -5.3352, -6.5123, -5.6315, -5.1959, -4.7164, -5.2327, -5.4266, -3.6287, -4.4069, -4.8544, -4.1315, -5.0852, -4.9164, -5.3482, -4.887, -4.0286, -5.502, -4.6215, -5.515, -5.1677, -5.5906, -4.6433, -5.635, -5.3507, -4.7466, -5.0799, -4.9621, -4.8681, -4.9582, -5.2843, -5.1573, -5.2584, -5.3154, -6.3287, -6.3506, -6.4652, -7.0409, -6.9103, -7.1848, -4.0338, -6.51, -5.9083, -6.953, -4.5591, -6.6736, -5.9634, -5.4639, -5.7418, -6.9239, -6.9068, -6.7202, -6.6857, -6.8961, -3.9496, -5.1734, -6.7847, -6.8238, -5.2966, -6.2123, -4.7347, -7.1277, -5.269, -4.8787, -4.607, -4.6636, -5.1698, -4.1241, -3.3095, -4.7622, -4.4448, -3.5145, -3.3701, -3.9736, -5.6605, -4.99, -5.8837, -5.5961, -4.079, -4.3385, -4.9498, -4.1756, -4.7511, -5.1616, -4.9322, -4.9564, -5.0158, -4.3577, -4.7404, -4.7031, -4.6138, -4.9714, -5.0931, -5.042, -5.3809, -5.6546, -5.6074, -5.4473, -5.8794, -6.0419, -5.7281, -6.1228, -5.7194, -6.0767, -6.2017, -4.6379, -4.9296, -6.6078, -5.3735, -6.6661, -6.7506, -6.6734, -5.5157, -5.0213, -5.3386, -6.1314, -5.5087, -6.8376, -6.7213, -6.0714, -5.8309, -6.2306, -6.7256, -6.0892, -5.2941, -5.6321, -5.47, -4.5865, -5.1588, -5.4647, -4.9496, -4.6308, -5.1596, -4.1923, -5.3984, -5.7782, -5.8223, -4.988, -5.398, -5.5686, -3.9208, -4.2561, -5.4503, -5.4218, -4.7543, -4.9809, -4.6706, -5.1636, -5.1587, -5.3747, -5.0862, -5.2334, -5.2749, -5.3597, -5.3867, -5.3887, -5.1474, -5.4169, -5.5142, -5.7593, -5.9088, -5.9553, -6.1287, -6.3066, -6.3834, -6.3898, -6.3985, -6.4818, -6.5761, -6.5761, -6.6036, -6.455, -6.6706, -6.6577, -6.7062, -6.6787, -6.8135, -6.0101, -6.883, -6.9278, -5.3468, -4.4799, -6.6254, -6.0588, -5.9798, -5.2862, -5.1759, -3.4207, -3.4273, -4.7926, -4.5754, -3.2263, -4.089, -5.5806, -4.1805, -5.3096, -4.8581, -4.7739, -5.046, -5.8297, -6.0118, -5.9914, -4.6049, -5.0638, -5.7259, -5.5299, -4.9674, -4.9022, -5.1548, -4.5652, -5.5454, -5.3044, -5.4544, -5.3366, -5.4653, -5.4745, -5.4842, -4.6597, -6.0236, -6.2637, -6.0371, -6.1017, -5.5218, -4.7697, -5.5447, -5.9441, -5.404, -5.8931, -2.7743, -5.4615, -5.9183, -6.2677, -4.8747, -6.459, -5.7794, -6.5144, -5.1877, -6.2901, -6.5844, -4.8275, -5.9871, -6.5127, -5.5783, -6.547, -5.6292, -4.8076, -5.8485, -4.4607, -5.1878, -4.1355, -5.0648, -5.5278, -5.141, -5.5131, -4.8001, -5.4365, -5.0292, -5.2504, -5.1972, -4.8133, -5.5084, -5.1532, -4.925, -4.6627, -5.3936, -4.924, -5.2181, -5.3459, -5.1159, -5.3361, -5.44, -5.3797, -5.4039, -5.4061, -6.2258, -4.5141, -4.6879, -6.0419, -4.9685, -4.9446, -4.4467, -5.3839, -5.9357, -4.7813, -5.3045, -5.9238, -6.2808, -5.9342, -5.3431, -5.9766, -4.6306, -6.1559, -5.8857, -6.6119, -6.3782, -6.6871, -5.669, -5.2185, -5.0642, -5.3136, -6.4937, -6.0342, -4.8845, -4.8823, -4.8317, -3.9365, -3.3756, -4.5449, -4.215, -4.7338, -4.4377, -3.5616, -4.0599, -5.1494, -4.6428, -4.5003, -4.6622, -3.9998, -4.4064, -4.5655, -4.586, -4.8805, -4.903, -4.9761, -4.8449, -4.9482, -4.6299, -4.9507, -5.0653, -5.0406, -5.2517, -3.5933, -5.7708, -5.6941, -5.3659, -5.97, -5.7526, -6.0058, -5.5762, -5.3399, -5.8347, -6.2197, -6.0071, -5.2609, -5.471, -5.4053, -5.9976, -5.7859, -5.2086, -5.6054, -5.5729, -5.8604, -6.1903, -4.9387, -5.8122, -6.2435, -5.7749, -5.1957, -5.3003, -5.3736, -4.8215, -4.7586, -3.9566, -4.9727, -5.5332, -5.397, -4.4799, -4.8859, -4.6711, -5.2717, -5.2121, -5.2613, -5.2447, -5.1417, -5.2816, -5.2159, -5.122, -5.1936, -5.1947, -4.162, -4.6434, -5.0625, -5.2046, -5.0921, -5.1803, -5.8366, -5.9941, -4.9278, -5.2552, -5.1421, -5.7502, -4.7974, -4.4248, -4.5242, -5.4207, -5.4841, -4.3389, -4.3223, -4.3123, -5.9616, -6.1186, -5.5847, -4.431, -5.2752, -5.9539, -4.9278, -4.7611, -5.1927, -5.8168, -5.1563, -4.9412, -4.5459, -4.7177, -4.5819, -4.9189, -4.7669, -5.0723, -4.4936, -4.9925, -4.8149, -4.7643, -4.8538, -5.0127, -4.9815, -5.1096, -5.103, -4.6496, -4.7081, -4.9397, -5.0803, -4.4045, -5.3199, -5.3817, -5.1529, -5.2754, -4.8355, -5.6658, -5.2025, -5.3973, -4.7844, -4.4812, -4.2667, -4.7398, -4.902, -5.2523, -5.5678, -5.3679, -4.5977, -5.4998, -4.3274, -5.3267, -5.4791, -5.3864, -4.1355, -5.0531, -5.5162, -4.2095, -3.9997, -4.4169, -4.1984, -5.1365, -4.4927, -4.9707, -4.5304, -4.6717, -4.9882, -5.1225, -5.1402, -5.2205, -5.1894, -5.1913, -3.342, -3.2796, -3.8925, -4.1061, -3.7813, -4.5622, -4.6492, -4.7531, -4.3029, -4.8836, -5.0851, -5.1035, -5.1291, -5.2894, -5.2854, -5.0742, -3.9179, -5.1321, -4.1105, -5.3791, -4.7884, -3.6144, -4.9534, -5.787, -5.1549, -5.7272, -4.9092, -6.0355, -6.3054, -5.7613, -4.7106, -3.7945, -4.7194, -4.4068, -4.6583, -4.29, -4.6572, -4.4723, -4.6351, -4.8253, -5.0182, -4.9991, -5.0393], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.7483, 1.7467, 1.7466, 1.7457, 1.7455, 1.7451, 1.7446, 1.7431, 1.742, 1.7411, 1.7357, 1.7344, 1.7226, 1.7225, 1.7041, 1.7019, 1.6994, 1.6849, 1.6208, 1.6058, 1.5798, 1.5769, 1.5704, 1.5684, 1.5638, 1.5521, 1.5142, 1.499, 1.4856, 1.4811, 1.4502, 1.367, 1.3091, 1.2382, 1.2722, 1.2795, 0.8588, 0.9971, 1.1038, 0.8529, 1.132, 1.0622, 1.2116, 0.9574, 0.5458, 1.194, 0.693, 1.1691, 0.9286, 1.1973, 0.3653, 1.2214, 0.7894, -0.0591, 0.3614, 0.0752, -0.3846, -0.6444, 0.349, -0.1757, 0.1857, 0.3686, 1.7712, 1.7705, 1.77, 1.7648, 1.7619, 1.7609, 1.7502, 1.7497, 1.7404, 1.7374, 1.6932, 1.6763, 1.6646, 1.6584, 1.6572, 1.6508, 1.648, 1.6467, 1.639, 1.6228, 1.6154, 1.6019, 1.6007, 1.5814, 1.5785, 1.5714, 1.5675, 1.5611, 1.5583, 1.5473, 1.5104, 1.4354, 1.4682, 1.3648, 1.1739, 1.3693, 1.3028, 1.06, 0.9437, 1.0636, 1.4458, 1.2833, 1.4811, 1.3902, 0.9054, 0.9467, 1.1578, 0.833, 0.971, 1.1768, 1.0479, 1.0611, 1.0065, 0.3297, 0.674, 0.6114, -0.1264, 0.4699, 0.7042, -0.0604, 1.9243, 1.9234, 1.9232, 1.9228, 1.9225, 1.9217, 1.9215, 1.9212, 1.921, 1.9206, 1.9204, 1.9199, 1.9191, 1.9176, 1.9176, 1.9167, 1.916, 1.9153, 1.9135, 1.9123, 1.9109, 1.9096, 1.9074, 1.9023, 1.897, 1.8968, 1.8925, 1.8912, 1.8886, 1.8866, 1.8673, 1.8742, 1.8618, 1.8339, 1.8446, 1.8585, 1.8109, 1.7223, 1.7627, 1.5837, 1.7542, 1.7951, 1.7963, 1.3937, 1.5931, 1.6452, 0.5666, 0.7255, 1.5301, 1.4882, 0.6898, 0.6525, 0.0168, 0.8151, 0.7895, 1.1881, -0.7723, 0.1705, -0.2663, 0.1463, 0.394, -0.0742, 1.9778, 1.977, 1.9758, 1.9758, 1.9751, 1.9747, 1.9738, 1.9726, 1.9719, 1.9719, 1.9719, 1.9709, 1.9703, 1.9703, 1.97, 1.9698, 1.9692, 1.969, 1.9689, 1.9684, 1.9677, 1.9665, 1.9662, 1.9662, 1.9535, 1.9446, 1.9254, 1.9246, 1.9245, 1.9225, 1.9151, 1.8946, 1.8942, 1.8721, 1.8059, 1.6815, 1.7557, 1.8287, 1.5119, 1.7193, 1.6134, 1.5236, 1.5617, 1.7851, 1.8407, 1.8242, 1.1808, 1.3892, 1.6964, 1.4961, 0.7163, 0.6038, 0.7998, -0.2513, 1.1143, 0.4763, 0.853, 0.3747, 0.7495, 0.6223, -0.0401, 2.4662, 2.4611, 2.4554, 2.4506, 2.4268, 2.4146, 2.3784, 2.3646, 2.3236, 2.2835, 2.2794, 2.264, 2.2342, 2.2328, 2.2044, 2.1666, 2.1559, 2.1525, 2.15, 2.1441, 2.1366, 2.0994, 2.0882, 2.0872, 2.0866, 2.0822, 2.0717, 2.016, 2.0046, 1.9934, 1.9179, 1.9427, 1.7913, 1.859, 1.9317, 1.8298, 1.8202, 1.3121, 1.6953, 1.3995, 1.5107, 1.2603, 0.5011, 1.6226, 0.987, 0.5811, -0.1752, 1.3183, -0.2366, 0.6077, 0.7509, -0.802, 0.3752, 1.2559, 0.2536, 0.2799, 0.3161, 2.5167, 2.5074, 2.5012, 2.4938, 2.4916, 2.488, 2.4654, 2.4295, 2.4207, 2.395, 2.3707, 2.3227, 2.3176, 2.1514, 2.1495, 2.1138, 2.1023, 2.0491, 2.0256, 2.0095, 2.0006, 1.994, 1.9732, 1.9492, 1.9218, 1.9048, 1.8675, 1.7951, 1.794, 1.791, 1.6631, 1.478, 1.3119, 1.4726, 1.3688, 1.481, 1.343, 0.9219, 0.8478, 1.5225, 1.0803, 0.941, 1.0855, 0.314, 0.632, 0.4431, 0.4512, 0.8119, 0.603, 0.8467, 0.1395, 0.337, 2.6298, 2.6292, 2.6287, 2.6283, 2.6258, 2.6249, 2.6243, 2.6241, 2.6227, 2.6227, 2.6215, 2.6204, 2.6037, 2.5651, 2.5614, 2.5512, 2.5394, 2.5266, 2.521, 2.4688, 2.4405, 2.4388, 2.424, 2.4034, 2.4021, 2.3763, 2.3345, 2.3102, 2.2954, 2.2807, 2.2638, 2.136, 2.1161, 2.1382, 1.6789, 1.2903, 0.3573, 1.3257, 1.9764, 1.7224, 0.0946, 0.7982, 0.3105, 1.4474, 1.2235, 1.3002, 1.1029, 0.4916, 1.0484, 0.4954, -0.4346, 0.1218, 0.1268, 2.7454, 2.744, 2.743, 2.7423, 2.7421, 2.7418, 2.7376, 2.736, 2.7257, 2.7167, 2.6835, 2.6632, 2.635, 2.6339, 2.6096, 2.5938, 2.584, 2.5676, 2.5645, 2.5525, 2.5517, 2.542, 2.5353, 2.5305, 2.5242, 2.515, 2.51, 2.4693, 2.4546, 2.4384, 2.4263, 2.3618, 2.232, 2.2633, 2.0914, 1.9748, 1.587, 1.7775, 0.488, 1.5079, 0.7689, -0.2768, -0.5399, 0.4314, -0.498, 1.0184, 0.5304, 3.1997, 3.1992, 3.1981, 3.1971, 3.1956, 3.1952, 3.1941, 3.1937, 3.1802, 3.1724, 3.1597, 3.1439, 3.1197, 3.1081, 3.0613, 3.06, 3.058, 3.0124, 3.0076, 3.0074, 3.003, 2.9888, 2.9759, 2.9267, 2.8199, 2.8066, 2.7138, 2.7011, 2.5992, 2.5844, 2.5026, 1.684, 1.9393, 1.5872, 2.4025, 1.3521, 2.1045, 1.162, 0.8343, 1.4153, 1.349, 0.6405, 2.0943, 1.0253, 1.107, 3.3988, 3.3987, 3.3977, 3.3968, 3.3963, 3.3951, 3.3945, 3.3939, 3.3937, 3.3931, 3.3913, 3.3912, 3.3908, 3.3895, 3.3893, 3.3524, 3.3498, 3.298, 3.29, 3.2897, 3.2817, 3.2368, 3.0466, 2.9603, 2.6493, 2.634, 2.6033, 2.4444, 2.44, 2.3187, 2.3019, 1.6468, 1.6958, 1.3154, 1.2176, 0.1974, 0.9266, 0.1022, 0.3493, 0.1833, 1.2891, -0.5156, 0.4667]}, \"token.table\": {\"Topic\": [1, 2, 5, 1, 2, 3, 4, 5, 7, 8, 1, 2, 3, 5, 4, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 3, 1, 2, 3, 4, 5, 7, 8, 10, 3, 4, 7, 3, 3, 3, 4, 7, 7, 9, 1, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 10, 1, 2, 5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 3, 7, 1, 3, 4, 8, 9, 2, 3, 4, 1, 2, 3, 4, 5, 6, 7, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 3, 4, 7, 8, 9, 4, 9, 4, 7, 1, 2, 3, 4, 5, 6, 7, 9, 1, 2, 3, 5, 6, 10, 2, 3, 4, 5, 7, 10, 1, 2, 5, 7, 9, 1, 2, 7, 8, 8, 10, 1, 2, 3, 5, 6, 9, 5, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 7, 4, 6, 7, 9, 1, 3, 4, 5, 6, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 10, 3, 4, 7, 4, 7, 9, 1, 2, 5, 6, 9, 1, 7, 1, 2, 3, 4, 5, 7, 10, 1, 2, 3, 4, 5, 9, 1, 2, 3, 4, 7, 9, 1, 2, 3, 5, 6, 10, 7, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 9, 1, 3, 1, 2, 3, 4, 7, 8, 10, 1, 3, 5, 7, 8, 10, 1, 2, 3, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 7, 8, 9, 7, 1, 2, 3, 5, 6, 7, 8, 9, 10, 1, 6, 7, 2, 9, 2, 3, 1, 2, 3, 5, 7, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 2, 4, 1, 2, 4, 6, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 6, 7, 8, 9, 4, 6, 7, 9, 2, 7, 8, 10, 1, 3, 4, 7, 9, 1, 2, 3, 5, 6, 1, 4, 8, 9, 6, 7, 7, 7, 8, 4, 7, 8, 9, 4, 1, 2, 3, 4, 5, 6, 7, 8, 9, 2, 3, 5, 4, 6, 8, 9, 1, 2, 3, 5, 6, 7, 8, 10, 1, 2, 3, 5, 6, 7, 8, 9, 2, 3, 5, 10, 1, 8, 9, 10, 2, 5, 7, 1, 3, 6, 1, 3, 5, 7, 8, 10, 1, 2, 3, 4, 5, 7, 10, 1, 2, 3, 5, 6, 7, 8, 10, 10, 6, 7, 1, 2, 3, 5, 6, 7, 10, 4, 1, 2, 3, 5, 6, 7, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 9, 10, 7, 2, 6, 8, 3, 1, 8, 9, 10, 2, 6, 1, 2, 3, 5, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 3, 6, 8, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 3, 4, 5, 6, 7, 8, 9, 10, 4, 6, 7, 9, 10, 1, 2, 5, 6, 9, 10, 4, 2, 6, 8, 10, 4, 4, 4, 6, 9, 5, 6, 10, 3, 7, 8, 2, 3, 4, 9, 1, 5, 7, 10, 4, 2, 3, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 7, 8, 1, 2, 5, 6, 10, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 6, 8, 2, 4, 5, 6, 7, 8, 1, 3, 6, 7, 9, 1, 9, 1, 2, 3, 4, 5, 4, 9, 5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 7, 9, 2, 3, 4, 6, 7, 8, 4, 7, 9, 1, 2, 3, 4, 5, 6, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 6, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 6, 7, 8, 1, 2, 3, 4, 6, 7, 2, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 7, 9, 3, 6, 1, 3, 4, 6, 7, 8, 9, 1, 3, 4, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 7, 8, 1, 2, 3, 4, 5, 6, 7, 9, 10, 1, 3, 4, 7, 9, 1, 4, 1, 8, 2, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 7, 1, 3, 7, 1, 4, 10, 7, 8, 1, 2, 3, 7, 1, 2, 3, 4, 5, 6, 7, 10, 1, 2, 7, 8, 8, 8, 10, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 6, 7, 10, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 6, 7, 4, 7, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 4, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 6, 7, 3, 10, 1, 4, 7, 4, 9, 2, 3, 6, 7, 8, 1, 2, 4, 6, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 5, 6, 7, 10, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 2, 6, 1, 3, 4, 5, 6, 1, 2, 5, 3, 5, 1, 3, 4, 5, 9, 1, 2, 3, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 7, 8, 1, 7, 4, 1, 2, 3, 4, 5, 7, 4, 1, 2, 3, 4, 5, 6, 7, 8, 10, 2, 3, 8, 1, 2, 3, 6, 1, 2, 3, 4, 6, 7, 8, 2, 4, 7, 9, 10, 6, 8, 7, 4, 6, 7, 7, 9, 7, 9, 2, 5, 8, 3, 1, 3, 4, 5, 7, 8, 10, 1, 3, 4, 6, 7, 8, 9, 4, 7, 9, 4, 1, 3, 5, 4, 1, 2, 3, 4, 5, 6, 7, 9, 5, 10, 6, 7, 10, 1, 2, 3, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 1, 2, 3, 4, 5, 6, 8, 10, 4, 1, 2, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 3, 2, 5, 7, 9, 1, 3, 1, 2, 3, 5, 10, 6, 7, 8, 10, 1, 5, 2, 3, 5, 1, 2, 3, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 3, 5, 2, 4, 5, 6, 1, 7, 8, 9, 1, 5, 5, 10, 1, 9, 1, 2, 3, 4, 5, 6, 7, 2, 3, 5, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 1, 5, 8, 8, 10, 1, 2, 3, 4, 7, 8, 9, 1, 7, 9, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 6, 7, 8, 10, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 2, 6, 6, 9, 3, 4, 8, 3, 5, 9, 1, 2, 3, 4, 5, 6, 7, 9, 1, 3, 7, 8, 9, 4, 1, 2, 3, 4, 5, 8, 1, 2, 9, 1, 2, 5, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 9, 10, 1, 2, 6, 7, 1, 1, 2, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 3, 1, 2, 7, 8, 1, 2, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 3, 4, 6, 7, 8, 9, 7, 9, 1, 2, 9, 1, 6, 8, 1, 2, 3, 4, 5, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 9, 10, 4, 1, 5, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 2, 3, 6, 7, 8, 10, 1, 2, 5, 6, 7, 9, 1, 2, 1, 2, 3, 5, 6, 7, 10, 4, 9, 1, 2, 1, 2, 3, 4, 6, 7, 2, 6, 7, 8, 9, 7, 10, 1, 2, 3, 5, 6, 7, 8, 9, 4, 7, 9, 2, 3, 4, 7, 2, 3, 4, 6, 7, 8, 9, 1, 2, 4, 5, 6, 1, 4, 5, 6, 1, 3, 5, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 7, 8, 1, 3, 4, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 9, 1, 2, 4, 6, 7, 8, 9, 10, 1, 2, 3, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 8, 8, 1, 2, 7, 1, 2, 3, 5, 6, 7, 8, 10, 2, 7, 3, 10, 3, 1, 4, 5, 1, 3, 5, 8, 2, 5, 10, 3, 6, 7, 8, 9, 1, 3, 4, 8, 1, 2, 3, 5, 6, 8, 10, 1, 3, 4, 5, 6, 7, 8, 9, 10, 4, 7, 9, 1, 2, 3, 4, 5, 6, 7, 4, 6, 7, 1, 2, 3, 5, 6, 7, 9, 10, 1, 2, 3, 5, 6, 9, 10, 1, 2, 5, 6, 10, 4, 6, 2, 7, 4, 5, 7, 9, 4, 7, 1, 3, 6, 1, 2, 2, 3, 5, 1, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 7, 9, 10, 1, 4, 6, 7, 8, 9, 4, 4, 7, 8, 3, 2, 3, 4, 6, 7, 9, 1, 2, 3, 4, 5, 6, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 2, 4, 6, 7, 8, 2, 1, 2, 1, 2, 3, 5, 6, 7, 10, 10, 1, 2, 3, 4, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 1, 6, 7, 8, 2, 2, 6, 8, 4, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 5, 6, 7, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 1, 3, 6, 3, 7, 3, 1, 3, 4, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 7, 9, 2, 3, 4, 5, 6, 7, 10, 8, 3, 5, 1, 2, 6, 7, 8, 10, 7, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 3, 4, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 5, 6, 7, 8, 9, 2, 3, 7, 8, 9, 10, 1, 3, 4, 5, 6, 4, 4, 2, 3, 9, 8, 1, 7, 8, 2, 5, 1, 2, 3, 4, 5, 8, 2, 3, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 2, 3, 5, 7, 8, 9, 10, 4, 7, 9, 10, 2, 8, 10, 1, 4, 7, 1, 3, 5, 9, 3, 5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 2, 3, 1, 4, 7, 8, 10, 4, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 9, 4, 6, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 4, 5, 7, 4, 6, 7, 4, 6, 7, 10], \"Freq\": [0.23859613925884293, 0.07953204641961431, 0.6703415341081778, 0.013245416488391401, 0.013245416488391401, 0.6733086714932296, 0.044151388294638, 0.09934062366293551, 0.0022075694147319002, 0.15673742844596492, 0.8269722989539368, 0.14785783307700323, 0.02063132554562836, 0.0017192771288023633, 0.9882338046666739, 0.25059471047902365, 0.21013725119686805, 0.11593779376378927, 0.024757549709975832, 0.04226898730971484, 0.10808783897769936, 0.15941446642521023, 0.03924977393044949, 0.029588291116800385, 0.01932296562729821, 0.0339297182477779, 0.9613420170203739, 0.9956658193848461, 0.13764743392213893, 0.0020242269694432198, 0.2186165126998677, 0.1133567102888203, 0.012145361816659317, 0.014169588786102538, 0.46152374903305404, 0.04048453938886439, 0.19924049040873296, 0.07218858348142498, 0.7247733781535068, 0.9911497184398467, 0.9938992753714226, 0.18223142728737263, 0.004237940169473782, 0.8094465723694924, 0.8924177723506427, 0.10320477639429201, 0.023710340443304875, 0.9721239581754999, 0.07590319651209876, 0.010469406415461898, 0.23032694114016175, 0.21331415571503617, 0.29706940703873136, 0.01963013702899106, 0.03140821924638569, 0.09160730613529161, 0.015704109623192845, 0.01439543382126011, 0.5869867397603901, 0.21841367060851724, 0.07215451618317087, 0.0643540279471524, 0.05655353971113393, 0.21825333710637526, 0.08487629776359039, 0.6911355675035217, 0.18689074685004958, 0.4898371689860014, 0.07020261556803893, 0.03921227175872445, 0.02276841585990452, 0.12522628722947485, 0.016443855898819932, 0.015178943906603013, 0.002846051982488065, 0.03130657180736871, 0.9952410115034402, 0.9901912138338667, 0.14211443941194152, 0.06767354257711501, 0.19625327347363353, 0.06090618831940351, 0.5346209863592085, 0.054212507990698613, 0.9216126358418764, 0.02258854499612442, 0.13951434739921725, 0.10907485342120621, 0.07609873494502759, 0.06087898795602207, 0.08624523293769794, 0.07609873494502759, 0.40332329520864624, 0.005073248996335172, 0.045659240967016554, 0.1887041537223989, 0.0408008981021403, 0.20230445308977898, 0.06460142199505547, 0.08330183362520312, 0.0017000374209225125, 0.30600673576605225, 0.0136002993673801, 0.09180202072981568, 0.00680014968369005, 0.004322319681576646, 0.4192650091129346, 0.012966959044729935, 0.017289278726306583, 0.5446122798786573, 0.1740268503624188, 0.8157508610738381, 0.17070715920903504, 0.8250846028436692, 0.09723420513049448, 0.10037079239276848, 0.028229285360466136, 0.08782444334367243, 0.018819523573644092, 0.04704880893411023, 0.608497928881159, 0.009409761786822046, 0.5735690309869939, 0.021850248799504528, 0.012745978466377641, 0.3131868994595649, 0.07647587079826584, 0.0018208540666253773, 0.14277600315760158, 0.0020692174370666896, 0.010346087185333449, 0.6807725367949409, 0.1572605252170684, 0.0062076523112000685, 0.015684076293381834, 0.21957706810734567, 0.7214675094955644, 0.034504967845440035, 0.006273630517352733, 0.09171375801874279, 0.05240786172499588, 0.006550982715624485, 0.8516277530311831, 0.9698393752035963, 0.02379973926879991, 0.015196316551111692, 0.015512906479259854, 0.0028493093533334423, 0.811736575771883, 0.15006362594222797, 0.004432258994074244, 0.9797511600121527, 0.9926430412708247, 0.2560476150109585, 0.16313890899269642, 0.2011802689371817, 0.0512095230021917, 0.08047210757487268, 0.043893876859021456, 0.039504489173119314, 0.036578230715851215, 0.014631292286340485, 0.11266095060482174, 0.9926058075267759, 0.7991316708530335, 0.042544720719771084, 0.0007090786786628514, 0.15670638798449016, 0.42099315249555735, 0.21640940142327808, 0.022468735666897725, 0.00827795524569916, 0.015373345456298442, 0.28263304338887135, 0.03429438601789653, 0.1530711288573946, 0.18298957677043082, 0.18716424392108702, 0.0507917836663173, 0.15515846243272272, 0.04870445009098919, 0.060532673684515134, 0.09880045589886378, 0.0034788892922135136, 0.060532673684515134, 0.24983905827766983, 0.3887409701368418, 0.11155175664796328, 0.003994966491524856, 0.03810583422685247, 0.12415126635200321, 0.0009219153441980437, 0.042408105833110006, 0.0003073051147326812, 0.03994966491524856, 0.6788593236833066, 0.008991514220970947, 0.04046181399436927, 0.006743635665728211, 0.05619696388106842, 0.020230906997184633, 0.13712059186980696, 0.04720544966009747, 0.9331041964507987, 0.009331041964507987, 0.052875904465545256, 0.04326343869882501, 0.008652687739765002, 0.9431429636343853, 0.031347412843385665, 0.020898275228923773, 0.6217236880604823, 0.29257585320493285, 0.036571981650616606, 0.9732449690475661, 0.022633603931338744, 0.03567632038836595, 0.03567632038836595, 0.023784213592243968, 0.15459738834958578, 0.725418514563441, 0.03567632038836595, 0.9988452194619684, 0.6422228607752953, 0.005489084280130729, 0.04528494531107852, 0.30052736433715743, 0.0027445421400653647, 0.0027445421400653647, 0.5975055349019823, 0.01107957283261954, 0.07122582535255419, 0.3070624470754558, 0.0031655922378912974, 0.008705378654201068, 0.03920868467937523, 0.1710924422372737, 0.01425770351977281, 0.581001418430742, 0.1354481834378417, 0.053466388199148035, 0.3117572252031339, 0.6801975822613832, 0.9959766921641436, 0.346310787571556, 0.3116797088144004, 0.13518637972672787, 0.0433931589246287, 0.13935879885409602, 0.005424144865578588, 0.016689676509472576, 0.0004172419127368144, 0.0008344838254736288, 0.0012517257382104433, 0.5371743604940711, 0.04424999521311314, 0.1306918463271016, 0.004116278624475641, 0.022639532434616024, 0.21610462778497114, 0.037046507620280764, 0.004116278624475641, 0.0010290696561189101, 0.0010290696561189101, 0.9934695690006636, 0.014446679279610587, 0.9852635268694421, 0.009699429562100856, 0.027158402773882394, 0.02327863094904205, 0.013579201386941197, 0.0737156646719665, 0.001939885912420171, 0.8496700296400349, 0.01471098042364948, 0.002942196084729896, 0.002942196084729896, 0.017653176508379376, 0.011768784338919584, 0.9503293353677564, 0.0037557977244823586, 0.17652249305067086, 0.06760435904068245, 0.015023190897929434, 0.6084392313661421, 0.12769712263240018, 0.32731527518569403, 0.03471525645908876, 0.05331271527645774, 0.039674578810387155, 0.2058118775788834, 0.0929872940868449, 0.03347542587126417, 0.007438983526947592, 0.16737712935632082, 0.038434748222562556, 0.04058318741663239, 0.8928301231659125, 0.062068404284261296, 0.9976002820517751, 0.5580998479879695, 0.15593966340840326, 0.09684673832732413, 0.02133911183483413, 0.039395283387386085, 0.019697641693693042, 0.037753813246245, 0.003282940282282174, 0.06730027578678456, 0.3066486288063409, 0.6866262775446329, 0.9844615806351831, 0.9768038446521521, 0.9903678236979986, 0.02271271547579423, 0.9652904077212547, 0.0768044622904373, 0.7335609867739726, 0.026646446100763964, 0.020376694077054794, 0.03291619812447313, 0.10972066041491044, 0.012064799777579339, 0.001723542825368477, 0.018958971079053245, 0.112030283648951, 0.08445359844305536, 0.6532227308146528, 0.022406056729790198, 0.001723542825368477, 0.09134776974452928, 0.9946010364058545, 0.9934304313905548, 0.821607521556415, 0.11517862451725445, 0.030714299871267853, 0.015357149935633927, 0.02303572490345089, 0.4386405637626624, 0.18968240595142158, 0.0811621833157525, 0.1614124319875078, 0.007295477151977753, 0.0009119346439972191, 0.007295477151977753, 0.021886431455933258, 0.023710300743927697, 0.066571229011797, 0.7418159627867734, 0.18614857493151243, 0.04306422255878273, 0.000555667387855261, 0.02833903678061831, 0.6259595201846874, 0.17962846153159803, 0.0639355541044671, 0.12969783832620468, 0.004327542596758606, 0.008655085193517213, 0.9780246268674451, 0.008655085193517213, 0.99413889760915, 0.07909611576248962, 0.8242647853143655, 0.09158497614604061, 0.004162953461183664, 0.2660675222716112, 0.07295399804221597, 0.004291411649542116, 0.6351289241322331, 0.012874234948626348, 0.736800600116522, 0.07073285761118611, 0.17093773922703312, 0.017683214402796528, 0.9899847872164382, 0.9970643225555536, 0.9959308142643104, 0.9005304770532357, 0.09895939308277316, 0.20969763222495053, 0.1149954757362632, 0.06087995774272757, 0.6087995774272758, 0.9966003505094836, 0.23354115941754378, 0.04106218187561209, 0.1334520910957393, 0.01796470457058029, 0.4285865233267012, 0.028230250039483314, 0.06159327281341814, 0.03592940914116058, 0.01796470457058029, 0.003011781777318302, 0.9908762047377213, 0.006023563554636604, 0.7693224916442271, 0.0834205111421451, 0.01390341852369085, 0.12976523955444794, 0.4089611146662508, 0.14891221310471245, 0.25658158287405114, 0.07080629740763289, 0.021533873953867733, 0.0038322996019595117, 0.04854246162482048, 0.040695371963665294, 0.062175885361735884, 0.6794936043103993, 0.024426240677824812, 0.01332340400608626, 0.048852481355649624, 0.14433687673260115, 0.024426240677824812, 0.0022205673343477102, 0.14028826614099843, 0.8146150580120161, 0.028293431826755984, 0.01532560890615949, 0.23774142152265668, 0.6796966281993903, 0.0731512066223559, 0.009143900827794487, 0.8739080133512269, 0.08525931837572946, 0.031972244390898547, 0.8400652778616903, 0.07074233918835286, 0.08842792398544108, 0.14832727162680298, 0.2768775737033656, 0.019776969550240397, 0.0692193934258414, 0.09888484775120199, 0.38565090622968773, 0.07737138089776766, 0.025790460299255885, 0.1547427617955353, 0.1547427617955353, 0.07737138089776766, 0.11605707134665148, 0.38685690448883825, 0.05963616953318865, 0.026504964236972732, 0.08172363973066593, 0.008834988078990911, 0.07288865165167502, 0.0883498807899091, 0.6162404185096161, 0.04859243443445001, 0.9976772317618043, 0.006049963394386652, 0.9921939966794109, 0.27624443610878474, 0.12794479146091084, 0.47688513180884945, 0.0785115765782862, 0.018900935102180008, 0.005815672339132311, 0.014539180847830777, 0.9935122515754851, 0.142889520785707, 0.4822521326517611, 0.17292879504179312, 0.017049317821021856, 0.010554339603489721, 0.072256632670045, 0.08687033365949232, 0.014613700989447306, 0.057597882595068484, 0.012671534170915067, 0.08524486624070135, 0.29720507419055336, 0.04147047546844931, 0.11865163814584108, 0.2050484620384438, 0.16012211361429038, 0.011519576519013697, 0.011519576519013697, 0.05386923285366329, 0.21034652828573283, 0.08465165162718516, 0.017956410951221097, 0.589996359825836, 0.025652015644601567, 0.005130403128920313, 0.010260806257840626, 0.9982946095761753, 0.009200808549861815, 0.9798861105602833, 0.009200808549861815, 0.9941085282101421, 0.12017385959550113, 0.8011590639700076, 0.07010141809737566, 0.010014488299625094, 0.3131511989654386, 0.6852485059714303, 0.07400030963585537, 0.24262396601919792, 0.5871499977664589, 0.06550847082518343, 0.03154111558249573, 0.020681986410232396, 0.6095743363015864, 0.055514805627465906, 0.008708204804308377, 0.03047871681507932, 0.09796730404846925, 0.06204595923069719, 0.10014435524954633, 0.01523935840753966, 0.0016238621722913526, 0.479039340825949, 0.5180120329609414, 0.989244360445056, 0.10448684053067006, 0.038594238394211465, 0.09036699721571465, 0.21368029549965858, 0.03482894684355669, 0.08848435144038726, 0.12896123560992612, 0.17696870288077451, 0.09036699721571465, 0.03200497818056561, 0.45151318976778304, 0.19988495999138903, 0.030478912339112513, 0.11766277786727156, 0.019846733616166286, 0.007088119148630817, 0.13254782807939627, 0.017720297871577043, 0.024099605105344778, 0.7520722150614305, 0.044643556145199076, 0.0034341197034768517, 0.10645771080778241, 0.08928711229039815, 0.41675438675249143, 0.16216123998151416, 0.05026998439426939, 0.3648627899584069, 0.0016216123998151416, 0.0016216123998151416, 0.9871710979541659, 0.03095550760402685, 0.8534875667967403, 0.10171095355608822, 0.013266646116011506, 0.9953430370865811, 0.9928086032266026, 0.9648174693259299, 0.0012661646579080444, 0.031654116447701115, 0.34203212747525835, 0.2067227144081232, 0.4510313768904506, 0.04139882247663712, 0.14489587866822992, 0.8072770382944239, 0.017063391218955455, 0.9794386559680431, 0.854913210429379, 0.13739676596186448, 0.23255025047393446, 0.07751675015797815, 0.03875837507898908, 0.6459729179831513, 0.9952897652198462, 0.010374083275922713, 0.9855379112126578, 0.002220020619721217, 0.1043409691268972, 0.05328049487330921, 0.05328049487330921, 0.0643805979719153, 0.5461250724514194, 0.0777007216902426, 0.09768090726773355, 0.019718116105741538, 0.8133722893618385, 0.0049295290264353845, 0.014788587079306154, 0.014788587079306154, 0.12816775468732, 0.06928087303301116, 0.9187709623762402, 0.0010658595851232486, 0.009592736266109236, 0.9852939878967332, 0.09611316850015209, 0.008009430708346007, 0.421296055259, 0.027232064408376423, 0.2883395055004563, 0.04164903968339924, 0.09290939621681368, 0.016018861416692013, 0.006407544566676806, 0.07108888659251934, 0.9038444152477461, 0.015233332841254147, 0.11713584205149007, 0.14109544610747668, 0.034608316969758426, 0.5590574279730207, 0.05856792102574503, 0.08518970331017459, 0.09294403480777087, 0.019363340584952267, 0.003872668116990453, 0.015490672467961812, 0.8674776582058615, 0.1685339664270585, 0.8258164354925868, 0.005244597406088186, 0.06293516887305824, 0.8417578836771539, 0.00786689610913228, 0.08129125979436688, 0.07184145228566335, 0.9236758151013859, 0.9936219065108092, 0.06965262546143662, 0.3421769755678343, 0.10414081865107999, 0.09061603700808259, 0.056127843818439216, 0.0973784278295813, 0.08182492894013428, 0.07100510362573635, 0.054099126571989606, 0.03313571502534363, 0.8655973430529508, 0.04121892109775956, 0.09274257246995901, 0.04688532279571143, 0.10046854884795306, 0.08707274233489265, 0.6630924223964901, 0.006697903256530204, 0.09377064559142285, 0.15870788114149414, 0.015870788114149416, 0.8173455878786949, 0.17060365730594454, 0.16934921864928318, 0.021325457163243067, 0.057704178206422416, 0.12293498835281297, 0.2747220658088371, 0.18189360521589673, 0.5226058518345389, 0.10483886389385887, 0.16917180310145408, 0.06512717302497292, 0.06512717302497292, 0.0015884676347554373, 0.027003949790842433, 0.003971169086888594, 0.04130015850364137, 0.15520173485956304, 0.4356184464073897, 0.11670522099100301, 0.0328233434037196, 0.03687560802146276, 0.11184250344971122, 0.05511079880130698, 0.024718814168233277, 0.010535888006132216, 0.020261323088715803, 0.042181565054308534, 0.8942491791513408, 0.050617878065170235, 0.008436313010861706, 0.04588438260986299, 0.8259188869775338, 0.12847627130761638, 0.0013660413278567818, 0.12430976083496713, 0.19534390988351977, 0.3852236544556124, 0.28960076150563774, 0.004098123983570345, 0.11630382262405053, 0.7753588174936702, 0.10855023444911382, 0.11349937538741571, 0.25279406336288046, 0.036113437623268636, 0.015477187552829416, 0.47979281413771185, 0.1083403128698059, 0.03703854765034824, 0.7963287744824872, 0.16050037315150903, 0.0894661695490974, 0.5377193489394204, 0.02305829112090139, 0.02305829112090139, 0.04427191895213067, 0.14665073152893285, 0.033203939214098005, 0.05072824046598306, 0.017524301251885057, 0.03412627085893406, 0.9089711067954676, 0.08455545179492721, 0.965239104333736, 0.027843435701934694, 0.15050860412355832, 0.03473275479774422, 0.2122557237639925, 0.03473275479774422, 0.03473275479774422, 0.08104309452806986, 0.4476666173931478, 0.006814662796364539, 0.1908105582982071, 0.013629325592729079, 0.7155395936182766, 0.07496129076000993, 0.006814662796364539, 0.3120042781272625, 0.06559180846993587, 0.04077328634617635, 0.015954764222416835, 0.382914341338004, 0.13650187168067734, 0.005318254740805611, 0.04077328634617635, 0.23553050977341652, 0.12315320772466223, 0.056958358572656284, 0.006927367934512251, 0.5056978592193944, 0.03001859438288642, 0.001539415096558278, 0.001539415096558278, 0.038485377413956946, 0.23442235282197452, 0.0389159637095768, 0.16863584274149945, 0.010192276209651066, 0.31318085080564184, 0.10099619153199692, 0.033356540322494396, 0.06022708669339266, 0.0009265705645137332, 0.04076910483860426, 0.9646723444296041, 0.00430657296620359, 0.02153286483101795, 0.00861314593240718, 0.16239310615901037, 0.027998811406725926, 0.10639548334555853, 0.005599762281345185, 0.027998811406725926, 0.6047743263852801, 0.033598573688071114, 0.02239904912538074, 0.005599762281345185, 0.06474732967826724, 0.01079122161304454, 0.06474732967826724, 0.8471108966239965, 0.00539561080652227, 0.9300661245629382, 0.05936592284444287, 0.9862724773085182, 0.9973546023915112, 0.8031330088245122, 0.07576726498344455, 0.11554507909975294, 0.003788363249172227, 0.0018941816245861135, 0.0030797923947713766, 0.006159584789542753, 0.010265974649237923, 0.2925802775032808, 0.18478754368628258, 0.35109633300393694, 0.0030797923947713766, 0.03490431380740894, 0.11292572114161714, 0.0010265974649237921, 0.14775668163178904, 0.011151447670323701, 0.7555105796644308, 0.08363585752742776, 0.951944249932663, 0.03146923140273266, 0.01573461570136633, 0.048150437135664304, 0.944952328787412, 0.996752972531523, 0.9259946413788047, 0.06614247438420033, 0.2141941615527961, 0.020080702645574634, 0.716211727692162, 0.04908616202251578, 0.18624166947043938, 0.0036280844702033647, 0.0205591453311524, 0.09433019622528749, 0.5744467077821994, 0.10884253410610094, 0.008465530430474518, 0.0024187229801355763, 0.2579969551487487, 0.0040312024241991986, 0.008062404848398397, 0.7256164363558557, 0.9888601483847795, 0.7568060086017118, 0.2409307518020431, 0.9988264441844872, 0.007075218638819356, 0.003537609319409678, 0.16828912905191756, 0.25268637995783416, 0.15110645521478483, 0.1455473548557125, 0.043462057352747475, 0.08085964158650694, 0.09349396058439864, 0.05306413979114517, 0.0723951478188999, 0.06205298384477134, 0.7446358061372561, 0.11376380371541413, 0.9743694639858325, 0.021978258586146597, 0.4695384730182487, 0.1446488784711223, 0.17248352070058115, 0.10175615110113649, 0.05247514518668474, 0.0036504448825519817, 0.023271586126268885, 0.0004563056103189977, 0.020990058074673894, 0.010495029037336947, 0.04470649230917647, 0.7655986807946471, 0.011176623077294117, 0.16485519039008822, 0.01303939359017647, 0.009651585837162184, 0.009651585837162184, 0.9748101695533806, 0.05072125576199116, 0.7937876526751616, 0.0012680313940497791, 0.05198928715604094, 0.006340156970248895, 0.04311306739769249, 0.03930897321554315, 0.007608188364298674, 0.003804094182149337, 0.998044506109109, 0.9874201414651252, 0.12997493396666746, 0.2995074565318859, 0.08916154890467043, 0.0646735178674722, 0.01695325225652184, 0.23483393866441366, 0.07597568603848677, 0.011302168171014561, 0.03202280981787459, 0.044580774452335215, 0.9906848162604703, 0.9177745434736492, 0.00042022643931943646, 0.08152392922797067, 0.09408184278532299, 0.9031856907391006, 0.003039597978380111, 0.9726713530816355, 0.021277185848660778, 0.9934301429058655, 0.9947704122245637, 0.7186113440223975, 0.277929822949429, 0.5349345960411994, 0.1876472143181171, 0.2744690597488877, 0.05368083276972434, 0.8857337407004516, 0.023485364336754397, 0.006710104096215542, 0.03019546843296994, 0.06706323449042614, 0.6649650293135212, 0.031170235749071306, 0.002833657795370119, 0.017001946772220715, 0.08312062866419015, 0.11051265401943464, 0.020780157166047537, 0.0009445525984567063, 0.0009445525984567063, 0.03209910490751337, 0.0035665672119459305, 0.9629731472254012, 0.293736341739351, 0.7037433187505284, 0.9909183751292566, 0.24565686760279629, 0.1044615651955816, 0.2795207815947705, 0.00573964643931767, 0.10847931770310397, 0.026976338264793048, 0.11766275200601224, 0.10905328234703573, 0.002295858575727068, 0.038164924261547294, 0.9610621836771455, 0.13841916723424022, 0.006591388915916201, 0.039548333495497204, 0.6789130583393687, 0.131827778318324, 0.17596167177599203, 0.08446160245247618, 0.7366928658354867, 0.9923925003460478, 0.005786545191522144, 0.008954046248382275, 0.008954046248382275, 0.0358161849935291, 0.8282492779753604, 0.11640260122896957, 0.38207598295092166, 0.06574374768980946, 0.08003586675281152, 0.22676828913296598, 0.1867503557565602, 0.03620670162627188, 0.005716847625200823, 0.008575271437801235, 0.009528079375334704, 0.24042881801816557, 0.07199280991504393, 0.06927610010692907, 0.0013583549040574326, 0.10187661780430744, 0.4210900202578041, 0.0027167098081148653, 0.08965142366779055, 0.09032254022211103, 0.05109153790341634, 0.07390026018172721, 0.2572823872993466, 0.1788203826619572, 0.20436615161366536, 0.005474093346794608, 0.09032254022211103, 0.007298791129059477, 0.04105570010095956, 0.060782560776629344, 0.9030551886813503, 0.008683222968089906, 0.026049668904269716, 0.9933819589799785, 0.9906979723988675, 0.9951219970008908, 0.004172120279145992, 0.018774541256156966, 0.07092604474548186, 0.08344240558291985, 0.542375636288979, 0.2816181188423545, 0.9958255030263221, 0.3370976560089181, 0.017436085655633697, 0.09492979968067235, 0.13948868524506958, 0.007749371402503865, 0.0019373428506259662, 0.01549874280500773, 0.3797191987226894, 0.009686714253129832, 0.018504619675075815, 0.05551385902522745, 0.9159786739162529, 0.010871188042973864, 0.8805662314808831, 0.03261356412892159, 0.06522712825784319, 0.1085904265882119, 0.13169477267081017, 0.016173042257818793, 0.616886040405374, 0.006931303824779483, 0.08548608050561363, 0.032346084515637585, 0.9910206914690701, 0.02198275277305209, 0.04396550554610418, 0.9067885518883987, 0.02198275277305209, 0.8143807994109468, 0.18010344602357475, 0.9851711086151419, 0.9179303129883314, 0.00041762070654610163, 0.08101841706994371, 0.1902105721744048, 0.805359656653331, 0.004102397335608267, 0.9927801552172005, 0.029340769971627153, 0.16626436317255386, 0.7922007892339331, 0.9947864375056645, 0.041061492822504674, 0.03041592060926272, 0.6585046811905378, 0.20378666808206022, 0.003041592060926272, 0.025853532517873312, 0.03649910473111526, 0.061047490882725486, 0.04114070037748891, 0.6927563095822327, 0.003981358101047314, 0.019906790505236572, 0.025215267973299656, 0.15660008530786101, 0.945358288494083, 0.016682793326366168, 0.033365586652732336, 0.9970593372504826, 0.7634133153384428, 0.021205925426067855, 0.20145629154764463, 0.9919963974191253, 0.27738202991196914, 0.038525281932217936, 0.007705056386443587, 0.1772162968882025, 0.4597350310578007, 0.010273408515258117, 0.012841760644072645, 0.017978464901701702, 0.004517964620288684, 0.9939522164635105, 0.5181081714153245, 0.008781494430768212, 0.46541920483071525, 0.0026107023992837613, 0.04960334558639146, 0.0026107023992837613, 0.9111351373500326, 0.02349632159355385, 0.013053511996418805, 0.7625011482065949, 0.009367335973053992, 0.0018734671946107984, 0.08805295814670752, 0.0018734671946107984, 0.1086610972874263, 0.013114270362275589, 0.009367335973053992, 0.005620401583832395, 0.1384013418920828, 0.09121906624705457, 0.3208394743861919, 0.1494105395425894, 0.07863712607504704, 0.07627801229279563, 0.06605518590303952, 0.061336958338536696, 0.010222826389756115, 0.007863712607504705, 0.9940975873674567, 0.5714152471073974, 0.033221816692290546, 0.06312145171535204, 0.0016610908346145273, 0.29567416856138584, 0.011627635842301691, 0.011627635842301691, 0.009966545007687164, 0.9849223278436811, 0.09413714220197711, 0.8879422332024327, 0.015265482519239532, 0.1633776824621811, 0.2351479604332893, 0.14799833718265792, 0.0020060015581986764, 0.06664382954460048, 0.2962195634273379, 0.04658381396261371, 0.02451779682242827, 0.00044577812404415036, 0.017385346837721864, 0.9950933847265521, 0.17815530221824047, 0.7126212088729619, 0.09376594853591604, 0.009376594853591604, 0.9693830279470014, 0.025510079682815823, 0.6188882986117799, 0.006861289341594011, 0.22916706400924, 0.14134256043683663, 0.004116773604956407, 0.016800845188245803, 0.020161014225894964, 0.06720338075298321, 0.8938049640146768, 0.00996642629755585, 0.9767097771604734, 0.0063199811290058465, 0.985917056124912, 0.0063199811290058465, 0.13490909367736967, 0.09743434543365587, 0.03372727341934242, 0.0037474748243713795, 0.19112121604294036, 0.5171515257632504, 0.022484848946228278, 0.2990163406530686, 0.48794211114558667, 0.04538996382749644, 0.01652353946351844, 0.006768678816381047, 0.006569600027663958, 0.07903427912068459, 0.020106957660426052, 0.0017917090984538066, 0.03702865470137867, 0.9893671309785901, 0.9747428220492981, 0.01282556344801708, 0.022265215156764585, 0.08906086062705834, 0.07792825304867605, 0.8126803532219073, 0.9822574754547931, 0.007673886526990571, 0.9905561458908874, 0.9904221081478158, 0.1672474027893365, 0.8216937615302184, 0.03750157597825638, 0.9562901874455377, 0.12997695819063135, 0.8648466833453547, 0.007192850971913247, 0.8397653509708715, 0.08991063714891558, 0.007192850971913247, 0.01258748920084818, 0.03416604211658792, 0.007192850971913247, 0.07187881731369268, 0.910465019306774, 0.01261031882696363, 0.0037830956480890887, 0.14536997440845353, 0.15912927239773103, 0.30061074954834527, 0.07089029616214709, 0.053541616088710255, 0.029313287020634664, 0.09810977696702214, 0.10439119561430099, 0.012861952468237658, 0.025723904936475316, 0.9924415270767719, 0.002553112354983341, 0.9931607060885197, 0.9962939111584722, 0.8567915323709034, 0.13658995443594113, 0.08439793972737362, 0.15529220909836747, 0.015191629150927251, 0.008439793972737362, 0.14178853874198769, 0.10127752767284835, 0.4962598855969569, 0.02808953716291576, 0.10299496959735778, 0.8645335326808518, 0.9864530876996955, 0.501027080054681, 0.0008107234305091925, 0.32915371278673217, 0.013782298318656272, 0.055129193274625087, 0.004053617152545963, 0.0324289372203677, 0.047021958969533165, 0.01621446861018385, 0.0008107234305091925, 0.0006036168599210432, 0.0030180842996052155, 0.09476784700760377, 0.2824926904430482, 0.11166911908539298, 0.09899316502705108, 0.05432551739289388, 0.10321848304649837, 0.21850930329141763, 0.033802544155578415, 0.01005202907707654, 0.016083246523322464, 0.036187304677475544, 0.07840582680119701, 0.8323080075819376, 0.022114463969568388, 0.9958238830968278, 0.11471270381868813, 0.0421097267182526, 0.1945759786291672, 0.1263291801547578, 0.0682467984744094, 0.13504153740681007, 0.2642748366455853, 0.023232952672139366, 0.008712357252052263, 0.024685012214148077, 0.9953252865582042, 0.8720721837891809, 0.11853408323348091, 0.027942885737669898, 0.9500581150807765, 0.1737551934267334, 0.0868775967133667, 0.7345105903948276, 0.070941326184119, 0.1064119892761785, 0.8158252511173685, 0.0788236117694361, 0.0012126709502990169, 0.010914038552691152, 0.8403809685572187, 0.012126709502990168, 0.0012126709502990169, 0.0024253419005980338, 0.05335752181315674, 0.006360946994354617, 0.06360946994354617, 0.6933432223846533, 0.20355030381934774, 0.03180473497177309, 0.9926015667105949, 0.15080280744685629, 0.05340932763742827, 0.14137763198142778, 0.012566900620571357, 0.5215263757537113, 0.11938555589542789, 0.9503529283302997, 0.02795155671559705, 0.019965397653997893, 0.17793423071562472, 0.02135210768587497, 0.7900279843773739, 0.00711736922862499, 0.07953741007589812, 0.03876613264203438, 0.052133764587563476, 0.44982081496705406, 0.02473011909922883, 0.04946023819845766, 0.09825209479963885, 0.003341907986382274, 0.19850933439110707, 0.006015434375488093, 0.11796645584828001, 0.5469353862056618, 0.03399033473594509, 0.006543593852909215, 0.20048844499330176, 0.026719674899379292, 0.038898030125627, 0.008543025307964807, 0.019994314550555933, 0.15238041704929045, 0.7424492660486705, 0.04214777492852714, 0.0616005941263089, 0.9967756432780416, 0.02195824084592503, 0.037642698593014334, 0.9379305732759405, 0.20852860405168383, 0.1130519539050131, 0.2902300161174916, 0.1325272905020952, 0.07885136280769822, 0.011400197032438296, 0.05177589485565726, 0.09880170761446523, 0.003325057467794503, 0.011875205242123226, 0.9972365442661163, 0.02382700089994098, 0.008664363963614903, 0.16029073332687568, 0.805785848616186, 0.8311717198077622, 0.04315338500130594, 0.12195521848195155, 0.029381823878942635, 0.23668691458037122, 0.06529294195320585, 0.07182223614852644, 0.03591111807426322, 0.47990312335606305, 0.0457050593672441, 0.02285252968362205, 0.003264647097660293, 0.009793941292980878, 0.24089110471426514, 0.7575009437400386, 0.037127180996832786, 0.03881478013305245, 0.010125594817318031, 0.34595782292503274, 0.31558103847307867, 0.20588709461879998, 0.03543958186061311, 0.001687599136219672, 0.010125594817318031, 0.0016099124135024532, 0.8983311267343689, 0.04024781033756133, 0.014489211721522078, 0.016099124135024533, 0.028978423443044157, 0.9348715150823151, 0.061211825392294444, 0.9886427397884497, 0.00887812595063393, 0.0019024555608501278, 0.9901635657577567, 0.5733940695581443, 0.42362696183773346, 0.07545439021452405, 0.024776068428649687, 0.21960606107212224, 0.3243412594295959, 0.09347334907172382, 0.032659362928674586, 0.10811375314319864, 0.001126184928574986, 0.12162797228609847, 0.13880106348441118, 0.33155469981824465, 0.09268775335669983, 0.0101449282280965, 0.3495388907680521, 0.06317523487496456, 0.0018445324051084542, 0.008300395822988044, 0.0036890648102169085, 0.9890167557166386, 0.047986095404300044, 0.8637497172774008, 0.05598377797168338, 0.023993047702150022, 0.030517170967254988, 0.09536615927267184, 0.09918080564357871, 0.09790925685327642, 0.34204662459131635, 0.1017239032241833, 0.11952558628841538, 0.07629292741813747, 0.029245622176952698, 0.008900841532116038, 0.12957963915910378, 0.4624214573913115, 0.25069002085682823, 0.007622331715241399, 0.04827476752986219, 0.0330301040993794, 0.04488706454531046, 0.021173143653448332, 0.0016938514922758665, 0.0847309059237109, 0.02523899325387133, 0.014422281859355046, 0.20191194603097065, 0.596721911930815, 0.07391419452919461, 0.09380817152588747, 0.8501700573717572, 0.01983372769404478, 0.03430698844375313, 0.0005360466944336426, 0.0005360466944336426, 0.18277683848996754, 0.8042180893558571, 0.09574178265521074, 0.034193493805432414, 0.24391358914541786, 0.01139783126847747, 0.02279566253695494, 0.25759098666759084, 0.3328166730395421, 0.9929152872157467, 0.9933324860867716, 0.1148089603349635, 0.8743143902431836, 0.1433947241313762, 0.0102424802950983, 0.0990106428526169, 0.0136566403934644, 0.1297380837379118, 0.5974780172140675, 0.9729832605586437, 0.011344797962617932, 0.004004046339747505, 0.006673410566245842, 0.004004046339747505, 0.9275171575130987, 0.06365313826070286, 0.026969172411170687, 0.4888162499524687, 0.05309555818449229, 0.04045375861675603, 0.3480708814316717, 0.010113439654189007, 0.029497532324717936, 0.003371146551396336, 0.128007373706768, 0.06892704738056739, 0.7975844054037083, 0.8533567961362967, 0.0514070359118251, 0.02056281436473004, 0.06168844309419012, 0.03397997205637202, 0.03155283119520259, 0.17718128286536838, 0.01941712688935544, 0.3810611152036005, 0.02184426775052487, 0.3325182979802119, 0.20055802979129206, 0.01114211276618289, 0.01114211276618289, 0.6796688787371564, 0.08913690212946312, 0.10861333965529181, 0.10289684809448697, 0.7259944282222136, 0.06288140716885315, 0.12637541894838442, 0.1699531496202411, 0.6754548254137788, 0.00435777306718567, 0.02178886533592835, 0.06460850025132403, 0.4466140988246865, 0.030736082643833762, 0.08656284499691957, 0.11604439365529073, 0.01568167481828253, 0.04390868949119109, 0.0037636019563878076, 0.06837210220771184, 0.12419886456079765, 0.0850396211066113, 0.710492318277817, 0.004572022640140392, 0.0009144045280280785, 0.0384049901771793, 0.06126510337788126, 0.010058449808308863, 0.01920249508858965, 0.021945708672673883, 0.04937784451351624, 0.2028274161055723, 0.1770343342886787, 0.046896512394351976, 0.09027578635912756, 0.08441372230983356, 0.07268959421124556, 0.2168963698238779, 0.02110343057745839, 0.04220686115491678, 0.04572409958449318, 0.01680607616642734, 0.8100528712217978, 0.15125468549784607, 0.011204050777618227, 0.0011204050777618227, 0.008963240622094582, 0.27178975837104763, 0.046403129477983744, 0.14583840693080605, 0.01988705549056446, 0.07291920346540302, 0.013258036993709641, 0.09280625895596749, 0.33807994333959585, 0.17061384775291674, 0.010036108691348043, 0.05687128258430558, 0.6333899707428543, 0.004460492751710242, 0.03233857244989925, 0.03122344926197169, 0.016726847818913405, 0.008920985503420483, 0.03456881882575437, 0.9865771396573604, 0.059199002776185866, 0.6222329965714318, 0.01093894616516478, 0.2361525436832632, 0.04504271950361968, 0.008365076479243656, 0.007078141636283093, 0.01093894616516478, 0.7888037255660467, 0.025174586986150427, 0.01678305799076695, 0.16783057990766953, 0.0716477617354091, 0.001302686577007438, 0.03386985100219339, 0.5536417952281611, 0.19279761339710083, 0.028659104694163638, 0.009118806039052066, 0.005210746308029752, 0.10291223958358761, 0.9949312933689575, 0.9975348917699788, 0.8392695032247265, 0.15494206213379566, 0.012911838511149638, 0.1301588287473879, 0.6617235110363385, 0.15599187872778553, 0.005961473072399447, 0.004471104804299585, 0.03576883843439668, 0.0024839471134997693, 0.0024839471134997693, 0.9758875638601004, 0.023802135703904888, 0.09555493979337192, 0.8958275605628617, 0.9957246337945768, 0.005742748995945849, 0.04594199196756679, 0.9418108353351193, 0.14713968950278525, 0.022636875308120807, 0.7243800098598658, 0.10186593888654363, 0.9641772656555623, 0.024409551029254743, 0.012204775514627371, 0.02298449390927804, 0.13024546548590887, 0.8274417807340094, 0.007661497969759346, 0.015322995939518693, 0.008600825953403494, 0.0645061946505262, 0.18061734502147336, 0.7439714449694023, 0.1093158659184791, 0.1552285296042403, 0.01311790391021749, 0.5269024737270692, 0.13992430837565323, 0.02623580782043498, 0.028422125138804565, 0.026123075105623842, 0.035452744786203784, 0.11941977191142328, 0.6269538025349722, 0.007463735744463955, 0.0018659339361159887, 0.07277142350852356, 0.0988944986141474, 0.013061537552811921, 0.9874936167208818, 0.02467232200929262, 0.9683886388647353, 0.11121751839450557, 0.07480702130106624, 0.7096736888030355, 0.02118428921800106, 0.06090483150175305, 0.0026480361522501325, 0.019860271141875992, 0.07787559711824397, 0.8754291262257772, 0.045651212103798196, 0.4067422786989654, 0.41814066114038184, 0.009898595278072167, 0.022496807450164014, 0.09178697439666919, 0.00029995743266885355, 0.0035994891920262426, 0.04739327436167886, 0.8763820416366157, 0.0023401389629816175, 0.023401389629816177, 0.030421806518761026, 0.015210903259380513, 0.0011700694814908087, 0.051483057185595586, 0.7744764044788969, 0.11063948635412812, 0.014751931513883751, 0.0073759657569418755, 0.08113562332636062, 0.37558297386210676, 0.6159560771338551, 0.2894086394339407, 0.7028495529109988, 0.08953166857433227, 0.041781445334688395, 0.7938474613590795, 0.07162533485946582, 0.9862922368967921, 0.005977528708465407, 0.01045565566323163, 0.02091131132646326, 0.9619203210173101, 0.1529505867350427, 0.8367296803740571, 0.07441348363833214, 0.8894183044391126, 0.033663242598293106, 0.11942441114121309, 0.08817316336594237, 0.055805799598697706, 0.16964963078004103, 0.07254753947830701, 0.2701000700576969, 0.06696695951843724, 0.12277275911713495, 0.03348347975921862, 0.04970688091668095, 0.04970688091668095, 0.003823606224360073, 0.04588327469232087, 0.23323997968596444, 0.6041297834488915, 0.011470818673080217, 0.1326257215493223, 0.015421595528990967, 0.095613892279744, 0.08327661585655122, 0.3392751016378013, 0.33002214432040666, 0.9863122915036375, 0.07107876197043765, 0.10661814295565647, 0.817405762660033, 0.9934925982014959, 0.1572113380415463, 0.02418635969869943, 0.13302497834284688, 0.5804726327687864, 0.04837271939739886, 0.03627953954804915, 0.6225161531018896, 0.010067646680893091, 0.2533691081358095, 0.005033823340446546, 0.001677941113482182, 0.06208382119884073, 0.043626468950536734, 0.005985904474454368, 0.011971808948908737, 0.21615766157751884, 0.22214356605197322, 0.0565335422587357, 0.3052811281971728, 0.021283215909171087, 0.08380266264236115, 0.0771516576707452, 0.008937462639691026, 0.026812387919073078, 0.5898725342196077, 0.008937462639691026, 0.36643596822733204, 0.9827864405515253, 0.9914169480815488, 0.004981994713977632, 0.2486840521656186, 0.27047351197441566, 0.02605261498877909, 0.07484205760412903, 0.2046314486391376, 0.002368419544434463, 0.17336831065260266, 0.9880624868692718, 0.004100681474721401, 0.018453066636246303, 0.03895647400985331, 0.05945988138346031, 0.014352385161524903, 0.8344886801058051, 0.004100681474721401, 0.024604088848328407, 0.08480841111966442, 0.5483502746367344, 0.04298508508804909, 0.05576443470882044, 0.06505850716029052, 0.13244053243344855, 0.04763212131378412, 0.0058087952821687956, 0.008132313395036315, 0.010455831507903832, 0.035838326331393236, 0.9616617565590518, 0.993741445592854, 0.9001742448046731, 0.07865600197322387, 0.017479111549605304, 0.9861248656075713, 0.22781771893869057, 0.5923260692405955, 0.17086328920401792, 0.9369327645599553, 0.061640313457891796, 0.21591408468261997, 0.0141583006349259, 0.20411550082018173, 0.1227052721693578, 0.0023597167724876502, 0.029496459656095625, 0.009438867089950601, 0.31384233074085743, 0.08848937896828687, 0.09139769735805851, 0.8186054632939154, 0.00198690646430562, 0.027816690500278678, 0.0099345323215281, 0.027816690500278678, 0.0099345323215281, 0.0099345323215281, 0.016662902588386493, 0.045822982118062856, 0.8477251691841629, 0.008331451294193246, 0.041657256470966235, 0.006248588470644936, 0.024994353882579742, 0.004165725647096623, 0.004165725647096623, 0.005291514042963585, 0.941889499647518, 0.05026938340815405, 0.8775979334730208, 0.11701305779640278, 0.9922266595772053, 0.19957098041538282, 0.6439764625774724, 0.09258447545043533, 0.0020574327877874517, 0.05760811805804865, 0.13402860166035574, 0.14023362951500185, 0.09493692617608532, 0.2010429024905336, 0.12285955152199277, 0.044055697767987304, 0.11789552923827588, 0.05832726183367333, 0.050881228408098014, 0.035368658771482764, 0.9430008279249515, 0.04715004139624758, 0.005547063693676186, 0.02515741764398827, 0.21635379173829913, 0.10566115410475073, 0.015094450586392963, 0.020125934115190616, 0.14591302233513198, 0.4729594517069795, 0.9954371751009675, 0.9927018224985306, 0.005805273815780881, 0.9822387745420598, 0.016508214698185877, 0.0010301304497094244, 0.9920156230701758, 0.0010301304497094244, 0.0061807826982565464, 0.002688642252427841, 0.9947976333983012, 0.1241847999408435, 0.39687925754290193, 0.026885369059357872, 0.001280255669493232, 0.0601720164661819, 0.08193636284756685, 0.2445288328732073, 0.057611505127195435, 0.003840767008479696, 0.07328462535734491, 0.0162854723016322, 0.0651418892065288, 0.10585556996060931, 0.0651418892065288, 0.6677043643669203, 0.10487775819435259, 0.10542399651828152, 0.02403448625287247, 0.11853371629257559, 0.08739813182862717, 0.31354079793519996, 0.019118341337512192, 0.1381982959540167, 0.0043699065914313585, 0.08412070188505365, 0.0020617476849829654, 0.008246990739931861, 0.9380951966672493, 0.0020617476849829654, 0.004123495369965931, 0.04742019675460821, 0.010406681851760278, 0.0034688939505867593, 0.04162672740704111, 0.8949746392513839, 0.05203340925880139, 0.0034688939505867593, 0.11735601160342338, 0.01805477101591129, 0.04513692753977822, 0.7898962319461189, 0.031595849277844756, 0.9856706136446723, 0.9975713649311634, 0.003735171678574075, 0.9487336063578151, 0.0448220601428889, 0.9923575997012345, 0.9462339586143818, 0.04628318275831215, 0.005142575862034683, 0.03114236764159442, 0.9550326076755622, 0.05506309560964325, 0.003933078257831661, 0.015732313031326644, 0.04326386083614827, 0.057029634738559085, 0.8220133558868171, 0.03594463978699731, 0.8746529014836012, 0.08387082616966039, 0.5827349237266429, 0.10871920218780652, 0.0884249511127493, 0.024643019162569477, 0.056533985137659386, 0.056533985137659386, 0.01884466171255313, 0.002899178725008174, 0.06088275322517165, 0.0034876773954010333, 0.0418521287448124, 0.04882748353561447, 0.0523151609310155, 0.7882150913606335, 0.0627781931172186, 0.0034876773954010333, 0.8610119186192949, 0.09491469969031598, 0.030508296329030137, 0.010169432109676711, 0.05654250348590186, 0.9355214213121945, 0.9935696025332398, 0.8411894586032507, 0.05670940170358993, 0.09451566950598322, 0.12754712863515041, 0.03924527034927706, 0.7652827718109025, 0.05886790552391558, 0.9677413698239891, 0.027649753423542544, 0.16856968000173186, 0.05965840526810264, 0.10197425086524521, 0.09711833415737639, 0.07145134584435549, 0.18591223967269194, 0.10821757234679083, 0.08393798880744674, 0.06382061958913306, 0.058964702881264236, 0.01275151069139178, 0.981866323237167, 0.004270990547480274, 0.995140797562904, 0.9936737048061773, 0.09779806328127451, 0.020589065953952528, 0.8698880365544942, 0.007720899732732198, 0.010223850754321107, 0.8179080603456885, 0.1533577613148166, 0.1078432963365894, 0.06522780020358229, 0.20264103263246233, 0.11306152035287598, 0.009566744029858737, 0.005218224016286584, 0.26091120081432917, 0.14611027245602434, 0.016524376051574183, 0.07392484023072661, 0.8712205541511857, 0.12144286512410467, 0.9447997746134943, 0.04246291121858402, 0.992863382816429, 0.0909717171709325, 0.43445514844533023, 0.0671932750773498, 0.10738651268069603, 0.03789209804590274, 0.10922742432665082, 0.10278423356580906, 0.03727846083058448, 0.008897739622114815, 0.00398864189956871, 0.011178568684882482, 0.08383926513661862, 0.005589284342441241, 0.8998747791330397, 0.005589284342441241, 0.016318448587830575, 0.9736674324072243, 0.008159224293915288, 0.9911986488314336, 0.05251285350038793, 0.05251285350038793, 0.8861544028190463], \"Term\": [\"abn\", \"abn\", \"abn\", \"access\", \"access\", \"access\", \"access\", \"access\", \"access\", \"access\", \"acquisition\", \"acquisition\", \"acquisition\", \"acquisition\", \"activist\", \"add\", \"add\", \"add\", \"add\", \"add\", \"add\", \"add\", \"add\", \"add\", \"add\", \"aerospace\", \"aerospace\", \"aerospatiale\", \"agreement\", \"agreement\", \"agreement\", \"agreement\", \"agreement\", \"agreement\", \"agreement\", \"agreement\", \"air\", \"air\", \"air\", \"airbus\", \"aircraft\", \"airline\", \"airline\", \"airline\", \"airport\", \"airport\", \"alcatel\", \"alcatel\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"allow\", \"amp\", \"amp\", \"amp\", \"amp\", \"amp\", \"amro\", \"amro\", \"amro\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"analyst\", \"aol\", \"apec\", \"appeal\", \"appeal\", \"appeal\", \"appeal\", \"appeal\", \"apple\", \"apple\", \"apple\", \"april\", \"april\", \"april\", \"april\", \"april\", \"april\", \"april\", \"april\", \"april\", \"area\", \"area\", \"area\", \"area\", \"area\", \"area\", \"area\", \"area\", \"area\", \"area\", \"army\", \"army\", \"army\", \"army\", \"army\", \"arrest\", \"arrest\", \"arrival\", \"arrival\", \"asia\", \"asia\", \"asia\", \"asia\", \"asia\", \"asia\", \"asia\", \"asia\", \"asset\", \"asset\", \"asset\", \"asset\", \"asset\", \"asset\", \"australia\", \"australia\", \"australia\", \"australia\", \"australia\", \"australia\", \"australian\", \"australian\", \"australian\", \"australian\", \"australian\", \"auto\", \"auto\", \"auto\", \"auto\", \"automaker\", \"automaker\", \"bank\", \"bank\", \"bank\", \"bank\", \"bank\", \"bank\", \"banking\", \"barrick\", \"base\", \"base\", \"base\", \"base\", \"base\", \"base\", \"base\", \"base\", \"base\", \"base\", \"bean\", \"beijing\", \"beijing\", \"beijing\", \"beijing\", \"bid\", \"bid\", \"bid\", \"bid\", \"bid\", \"bid\", \"bid\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"billion\", \"board\", \"board\", \"board\", \"board\", \"board\", \"board\", \"board\", \"board\", \"boeing\", \"boeing\", \"boeing\", \"bomb\", \"bomb\", \"bomb\", \"bond\", \"bond\", \"bond\", \"bond\", \"bond\", \"bonus\", \"bonus\", \"branch\", \"branch\", \"branch\", \"branch\", \"branch\", \"branch\", \"bre\", \"britain\", \"britain\", \"britain\", \"britain\", \"britain\", \"britain\", \"british\", \"british\", \"british\", \"british\", \"british\", \"british\", \"brokerage\", \"brokerage\", \"brokerage\", \"brokerage\", \"brokerage\", \"brokerage\", \"bus\", \"bus\", \"busang\", \"business\", \"business\", \"business\", \"business\", \"business\", \"business\", \"business\", \"business\", \"business\", \"business\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"buy\", \"bzw\", \"cable\", \"cable\", \"canada\", \"canada\", \"canada\", \"canada\", \"canada\", \"canada\", \"canada\", \"canadian\", \"canadian\", \"canadian\", \"canadian\", \"canadian\", \"canadian\", \"capacity\", \"capacity\", \"capacity\", \"capacity\", \"capacity\", \"capacity\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"capital\", \"car\", \"car\", \"car\", \"cargo\", \"cash\", \"cash\", \"cash\", \"cash\", \"cash\", \"cash\", \"cash\", \"cash\", \"cash\", \"casino\", \"casino\", \"cathay\", \"cba\", \"ce\", \"cellular\", \"cellular\", \"cent\", \"cent\", \"cent\", \"cent\", \"cent\", \"cent\", \"central\", \"central\", \"central\", \"central\", \"central\", \"central\", \"central\", \"central\", \"central\", \"cereal\", \"chee\", \"chemical\", \"chemical\", \"chemical\", \"chemical\", \"chemical\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"chief\", \"china\", \"china\", \"china\", \"china\", \"china\", \"chinese\", \"chinese\", \"chinese\", \"chinese\", \"chrysler\", \"chrysler\", \"chrysler\", \"chrysler\", \"cigarette\", \"civil\", \"civil\", \"civil\", \"civil\", \"client\", \"client\", \"client\", \"client\", \"client\", \"club\", \"club\", \"club\", \"club\", \"cnb\", \"coast\", \"cocoa\", \"coffee\", \"coffee\", \"colombia\", \"colombia\", \"colombia\", \"colombia\", \"colony\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"commercial\", \"communications\", \"communications\", \"communications\", \"communist\", \"communist\", \"communist\", \"communist\", \"company\", \"company\", \"company\", \"company\", \"company\", \"company\", \"company\", \"company\", \"compare\", \"compare\", \"compare\", \"compare\", \"compare\", \"compare\", \"compare\", \"compare\", \"computer\", \"computer\", \"computer\", \"computer\", \"conrail\", \"conrail\", \"conrail\", \"conrail\", \"consensus\", \"consensus\", \"consensus\", \"consolidation\", \"consolidation\", \"consolidation\", \"consultant\", \"consultant\", \"consultant\", \"consultant\", \"consultant\", \"consultant\", \"contain\", \"contain\", \"contain\", \"contain\", \"contain\", \"contain\", \"contain\", \"contract\", \"contract\", \"contract\", \"contract\", \"contract\", \"contract\", \"contract\", \"contract\", \"copper\", \"corn\", \"corn\", \"corp\", \"corp\", \"corp\", \"corp\", \"corp\", \"corp\", \"corp\", \"corruption\", \"cost\", \"cost\", \"cost\", \"cost\", \"cost\", \"cost\", \"cost\", \"cost\", \"country\", \"country\", \"country\", \"country\", \"country\", \"country\", \"country\", \"country\", \"country\", \"country\", \"credit\", \"credit\", \"credit\", \"credit\", \"credit\", \"credit\", \"credit\", \"credit\", \"crop\", \"crown\", \"crown\", \"crown\", \"csf\", \"csx\", \"csx\", \"csx\", \"csx\", \"currency\", \"currency\", \"customer\", \"customer\", \"customer\", \"customer\", \"customer\", \"cut\", \"cut\", \"cut\", \"cut\", \"cut\", \"cut\", \"cut\", \"cut\", \"cut\", \"czech\", \"czech\", \"czech\", \"dassault\", \"day\", \"day\", \"day\", \"day\", \"day\", \"day\", \"day\", \"day\", \"day\", \"day\", \"deal\", \"deal\", \"deal\", \"deal\", \"deal\", \"deal\", \"deal\", \"deal\", \"deal\", \"death\", \"death\", \"death\", \"death\", \"death\", \"debt\", \"debt\", \"debt\", \"debt\", \"debt\", \"debt\", \"declaration\", \"deficit\", \"deficit\", \"deficit\", \"deficit\", \"democracy\", \"democratic\", \"deng\", \"deng\", \"deng\", \"deposit\", \"deposit\", \"deposit\", \"detroit\", \"detroit\", \"detroit\", \"digital\", \"digital\", \"diplomat\", \"diplomat\", \"discovery\", \"discovery\", \"discovery\", \"discovery\", \"dissident\", \"distance\", \"distance\", \"domestic\", \"domestic\", \"domestic\", \"domestic\", \"domestic\", \"domestic\", \"domestic\", \"domestic\", \"earn\", \"earn\", \"earn\", \"earn\", \"earn\", \"earn\", \"earnings\", \"earnings\", \"earnings\", \"earnings\", \"eaton\", \"economic\", \"economic\", \"economic\", \"economic\", \"economic\", \"economic\", \"economic\", \"economic\", \"economic\", \"economist\", \"economist\", \"economist\", \"economy\", \"economy\", \"economy\", \"economy\", \"economy\", \"economy\", \"electric\", \"electric\", \"electric\", \"electric\", \"electric\", \"electricity\", \"electricity\", \"electronic\", \"electronic\", \"electronic\", \"electronic\", \"electronic\", \"embassy\", \"embassy\", \"encryption\", \"end\", \"end\", \"end\", \"end\", \"end\", \"end\", \"end\", \"end\", \"end\", \"end\", \"england\", \"england\", \"england\", \"enterprise\", \"enterprise\", \"enterprise\", \"enterprise\", \"enterprise\", \"enterprise\", \"ethnic\", \"ethnic\", \"ethnic\", \"exchange\", \"exchange\", \"exchange\", \"exchange\", \"exchange\", \"exchange\", \"exchange\", \"executive\", \"executive\", \"executive\", \"executive\", \"executive\", \"executive\", \"executive\", \"executive\", \"executive\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expect\", \"expectation\", \"expectation\", \"expectation\", \"expectation\", \"expense\", \"expense\", \"expense\", \"export\", \"export\", \"export\", \"export\", \"export\", \"export\", \"exporter\", \"exporter\", \"exporter\", \"factor\", \"factor\", \"factor\", \"factor\", \"factor\", \"factor\", \"factory\", \"factory\", \"factory\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"fall\", \"farmer\", \"farmer\", \"fcc\", \"fcc\", \"fight\", \"fight\", \"fight\", \"fight\", \"fight\", \"fight\", \"fight\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"finance\", \"finance\", \"finance\", \"finance\", \"finance\", \"finance\", \"finance\", \"finance\", \"financial\", \"financial\", \"financial\", \"financial\", \"financial\", \"financial\", \"financial\", \"financial\", \"financial\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"firm\", \"fiscal\", \"fiscal\", \"fiscal\", \"fiscal\", \"fix\", \"fix\", \"fix\", \"fix\", \"fix\", \"fix\", \"fix\", \"fix\", \"fix\", \"flight\", \"flight\", \"flight\", \"flight\", \"flight\", \"florida\", \"florida\", \"flotation\", \"ford\", \"forecast\", \"forecast\", \"forecast\", \"forecast\", \"forecast\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"foreign\", \"france\", \"france\", \"france\", \"france\", \"franchise\", \"franchise\", \"franchise\", \"freedom\", \"freedom\", \"freeport\", \"freight\", \"freight\", \"french\", \"french\", \"french\", \"french\", \"fund\", \"fund\", \"fund\", \"fund\", \"fund\", \"fund\", \"fund\", \"fund\", \"gas\", \"gas\", \"gas\", \"gas\", \"gazprom\", \"gm\", \"gm\", \"gold\", \"government\", \"government\", \"government\", \"government\", \"government\", \"government\", \"government\", \"government\", \"government\", \"government\", \"grade\", \"grade\", \"grade\", \"grade\", \"grain\", \"grain\", \"group\", \"group\", \"group\", \"group\", \"group\", \"group\", \"group\", \"group\", \"group\", \"group\", \"growth\", \"growth\", \"growth\", \"growth\", \"growth\", \"guerrilla\", \"guerrilla\", \"guerrilla\", \"half\", \"half\", \"half\", \"half\", \"half\", \"half\", \"half\", \"half\", \"half\", \"handover\", \"harvest\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"hilton\", \"hong\", \"hong\", \"hong\", \"hughes\", \"hughes\", \"human\", \"human\", \"human\", \"hwa\", \"hwang\", \"ibm\", \"ibm\", \"import\", \"import\", \"import\", \"income\", \"income\", \"income\", \"income\", \"income\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"increase\", \"index\", \"index\", \"index\", \"indonesia\", \"indonesia\", \"indonesian\", \"industry\", \"industry\", \"industry\", \"industry\", \"industry\", \"industry\", \"industry\", \"industry\", \"industry\", \"inflation\", \"inflation\", \"institution\", \"institution\", \"institution\", \"institution\", \"institution\", \"insurance\", \"insurance\", \"insurance\", \"internet\", \"internet\", \"investigation\", \"investigation\", \"investigation\", \"investigation\", \"investigation\", \"investment\", \"investment\", \"investment\", \"investment\", \"investment\", \"investment\", \"investment\", \"investment\", \"investment\", \"investor\", \"investor\", \"investor\", \"investor\", \"investor\", \"investor\", \"investor\", \"investor\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"issue\", \"item\", \"item\", \"item\", \"item\", \"itt\", \"ivory\", \"jail\", \"japan\", \"japan\", \"japan\", \"japan\", \"japan\", \"japan\", \"jiang\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"johnson\", \"johnson\", \"johnson\", \"jones\", \"jones\", \"jones\", \"jones\", \"july\", \"july\", \"july\", \"july\", \"july\", \"july\", \"july\", \"kellogg\", \"kill\", \"kill\", \"kill\", \"kill\", \"klaus\", \"klaus\", \"kmart\", \"kong\", \"kong\", \"kong\", \"korea\", \"korea\", \"korean\", \"korean\", \"labor\", \"labor\", \"labor\", \"lagardere\", \"law\", \"law\", \"law\", \"law\", \"law\", \"law\", \"law\", \"leader\", \"leader\", \"leader\", \"leader\", \"leader\", \"leader\", \"leader\", \"lee\", \"lee\", \"lee\", \"legislature\", \"liability\", \"liability\", \"liability\", \"liberty\", \"life\", \"life\", \"life\", \"life\", \"life\", \"life\", \"life\", \"life\", \"lloyd\", \"lloyd\", \"lme\", \"lme\", \"lme\", \"loan\", \"loan\", \"loan\", \"loan\", \"loan\", \"loan\", \"london\", \"london\", \"london\", \"london\", \"london\", \"london\", \"london\", \"london\", \"london\", \"long\", \"long\", \"long\", \"long\", \"long\", \"long\", \"long\", \"long\", \"long\", \"long\", \"mainframe\", \"management\", \"management\", \"management\", \"management\", \"management\", \"management\", \"management\", \"management\", \"mao\", \"margin\", \"margin\", \"margin\", \"market\", \"market\", \"market\", \"market\", \"market\", \"market\", \"market\", \"market\", \"market\", \"market\", \"mci\", \"melbourne\", \"melbourne\", \"melbourne\", \"melbourne\", \"mercury\", \"mercury\", \"merger\", \"merger\", \"merger\", \"merger\", \"merger\", \"metal\", \"metal\", \"metal\", \"metal\", \"mgam\", \"mgam\", \"microsoft\", \"microsoft\", \"microsoft\", \"mid\", \"mid\", \"mid\", \"mid\", \"mid\", \"mid\", \"mid\", \"million\", \"million\", \"million\", \"million\", \"million\", \"million\", \"million\", \"million\", \"million\", \"million\", \"minerals\", \"modem\", \"modem\", \"monetary\", \"monetary\", \"monetary\", \"monetary\", \"morris\", \"morris\", \"moscow\", \"moslem\", \"mutual\", \"mutual\", \"names\", \"names\", \"natwest\", \"natwest\", \"net\", \"net\", \"net\", \"net\", \"net\", \"net\", \"net\", \"network\", \"network\", \"network\", \"network\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"nickel\", \"nomura\", \"nomura\", \"norfolk\", \"norilsk\", \"norilsk\", \"north\", \"north\", \"north\", \"north\", \"north\", \"north\", \"north\", \"northern\", \"northern\", \"northern\", \"nynex\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"offer\", \"official\", \"official\", \"official\", \"official\", \"official\", \"official\", \"official\", \"official\", \"official\", \"official\", \"oil\", \"oil\", \"oil\", \"oil\", \"oil\", \"oil\", \"online\", \"open\", \"open\", \"open\", \"open\", \"open\", \"open\", \"open\", \"open\", \"open\", \"open\", \"ounce\", \"outlook\", \"outlook\", \"packer\", \"packer\", \"pact\", \"pact\", \"pact\", \"panel\", \"panel\", \"panel\", \"party\", \"party\", \"party\", \"party\", \"party\", \"party\", \"party\", \"party\", \"passenger\", \"passenger\", \"passenger\", \"passenger\", \"passenger\", \"patten\", \"payment\", \"payment\", \"payment\", \"payment\", \"payment\", \"payment\", \"penny\", \"penny\", \"penny\", \"pension\", \"pension\", \"pension\", \"pension\", \"people\", \"people\", \"people\", \"people\", \"people\", \"people\", \"people\", \"people\", \"people\", \"people\", \"percent\", \"percent\", \"percent\", \"percent\", \"percent\", \"percent\", \"percent\", \"percent\", \"percent\", \"performance\", \"performance\", \"performance\", \"performance\", \"philip\", \"phone\", \"phone\", \"phone\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plan\", \"plane\", \"plant\", \"plant\", \"plant\", \"plant\", \"plc\", \"plc\", \"plc\", \"point\", \"point\", \"point\", \"point\", \"point\", \"point\", \"point\", \"point\", \"point\", \"point\", \"police\", \"police\", \"policy\", \"policy\", \"policy\", \"policy\", \"policy\", \"policy\", \"policy\", \"policy\", \"policy\", \"political\", \"political\", \"political\", \"political\", \"political\", \"political\", \"port\", \"port\", \"pound\", \"pound\", \"pound\", \"pounds\", \"prague\", \"prague\", \"president\", \"president\", \"president\", \"president\", \"president\", \"president\", \"president\", \"president\", \"president\", \"price\", \"price\", \"price\", \"price\", \"price\", \"price\", \"price\", \"price\", \"price\", \"prison\", \"probe\", \"probe\", \"probe\", \"probe\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"problem\", \"product\", \"product\", \"product\", \"product\", \"product\", \"product\", \"product\", \"product\", \"product\", \"production\", \"production\", \"production\", \"production\", \"production\", \"production\", \"profit\", \"profit\", \"profit\", \"profit\", \"profit\", \"profit\", \"profitability\", \"profitability\", \"project\", \"project\", \"project\", \"project\", \"project\", \"project\", \"project\", \"provisional\", \"pyongyang\", \"quaker\", \"quaker\", \"quality\", \"quality\", \"quality\", \"quality\", \"quality\", \"quality\", \"quarter\", \"quarter\", \"quarter\", \"quarter\", \"quarter\", \"rain\", \"rain\", \"rate\", \"rate\", \"rate\", \"rate\", \"rate\", \"rate\", \"rate\", \"rate\", \"rebel\", \"rebel\", \"rebel\", \"reduction\", \"reduction\", \"reduction\", \"reduction\", \"region\", \"region\", \"region\", \"region\", \"region\", \"region\", \"region\", \"regulate\", \"regulate\", \"regulate\", \"regulate\", \"regulate\", \"regulation\", \"regulation\", \"regulation\", \"regulation\", \"regulator\", \"regulator\", \"regulator\", \"regulator\", \"regulator\", \"report\", \"report\", \"report\", \"report\", \"report\", \"report\", \"report\", \"report\", \"report\", \"report\", \"result\", \"result\", \"result\", \"result\", \"result\", \"result\", \"result\", \"result\", \"result\", \"result\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"reuters\", \"revenue\", \"revenue\", \"revenue\", \"revenue\", \"revenue\", \"revenue\", \"rich\", \"rich\", \"rich\", \"rich\", \"rich\", \"rich\", \"rich\", \"rich\", \"right\", \"right\", \"right\", \"right\", \"right\", \"right\", \"right\", \"right\", \"right\", \"right\", \"riot\", \"rise\", \"rise\", \"rise\", \"rise\", \"rise\", \"rise\", \"rise\", \"rise\", \"royal\", \"royal\", \"royal\", \"royal\", \"rule\", \"rule\", \"rule\", \"rule\", \"rule\", \"rule\", \"rule\", \"rule\", \"rule\", \"russia\", \"russian\", \"sainsbury\", \"sainsbury\", \"sainsbury\", \"sale\", \"sale\", \"sale\", \"sale\", \"sale\", \"sale\", \"sale\", \"sale\", \"sales\", \"sales\", \"sample\", \"sample\", \"satellite\", \"scandal\", \"scandal\", \"scandal\", \"scheme\", \"scheme\", \"scheme\", \"scheme\", \"sears\", \"sears\", \"sears\", \"season\", \"season\", \"season\", \"season\", \"season\", \"seat\", \"seat\", \"seat\", \"seat\", \"securities\", \"securities\", \"securities\", \"securities\", \"securities\", \"securities\", \"securities\", \"security\", \"security\", \"security\", \"security\", \"security\", \"security\", \"security\", \"security\", \"security\", \"sentence\", \"seoul\", \"seoul\", \"service\", \"service\", \"service\", \"service\", \"service\", \"service\", \"service\", \"shanghai\", \"shanghai\", \"shanghai\", \"share\", \"share\", \"share\", \"share\", \"share\", \"share\", \"share\", \"share\", \"shareholder\", \"shareholder\", \"shareholder\", \"shareholder\", \"shareholder\", \"shareholder\", \"shareholder\", \"shares\", \"shares\", \"shares\", \"shares\", \"shares\", \"shenzhen\", \"shenzhen\", \"shipment\", \"shipment\", \"singapore\", \"singapore\", \"singapore\", \"singapore\", \"sino\", \"sino\", \"skoda\", \"skoda\", \"skoda\", \"snapple\", \"snapple\", \"software\", \"software\", \"software\", \"source\", \"source\", \"source\", \"source\", \"source\", \"source\", \"source\", \"source\", \"source\", \"south\", \"south\", \"south\", \"south\", \"south\", \"south\", \"south\", \"southern\", \"southern\", \"southern\", \"southern\", \"southern\", \"southern\", \"sovereignty\", \"soviet\", \"soviet\", \"soviet\", \"sprint\", \"stable\", \"stable\", \"stable\", \"stable\", \"stable\", \"stable\", \"stake\", \"stake\", \"stake\", \"stake\", \"stake\", \"stake\", \"stake\", \"state\", \"state\", \"state\", \"state\", \"state\", \"state\", \"state\", \"state\", \"state\", \"steel\", \"steel\", \"steel\", \"steel\", \"steel\", \"sterling\", \"stg\", \"stg\", \"stock\", \"stock\", \"stock\", \"stock\", \"stock\", \"stock\", \"stock\", \"strathcona\", \"strike\", \"strike\", \"strike\", \"strike\", \"strike\", \"strike\", \"strike\", \"strike\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"strong\", \"subscriber\", \"subscriber\", \"suez\", \"sugar\", \"sugar\", \"sugar\", \"surcharge\", \"surplus\", \"surplus\", \"surplus\", \"taiwan\", \"taiwan\", \"talk\", \"talk\", \"talk\", \"talk\", \"talk\", \"talk\", \"talk\", \"talk\", \"talk\", \"tax\", \"tax\", \"tax\", \"tax\", \"tax\", \"tax\", \"tax\", \"tax\", \"technology\", \"technology\", \"technology\", \"technology\", \"technology\", \"technology\", \"technology\", \"technology\", \"technology\", \"telecom\", \"telecom\", \"telecom\", \"telecommunication\", \"telecommunication\", \"telecommunications\", \"television\", \"television\", \"television\", \"television\", \"television\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"tell\", \"territory\", \"territory\", \"territory\", \"test\", \"test\", \"test\", \"test\", \"test\", \"test\", \"test\", \"textile\", \"thomson\", \"thomson\", \"tobacco\", \"tobacco\", \"tonne\", \"tonne\", \"tonne\", \"tonne\", \"toronto\", \"toronto\", \"total\", \"total\", \"total\", \"total\", \"total\", \"total\", \"total\", \"total\", \"total\", \"town\", \"town\", \"town\", \"town\", \"town\", \"town\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trade\", \"trader\", \"trader\", \"trader\", \"trader\", \"trader\", \"trader\", \"truck\", \"truck\", \"truck\", \"truck\", \"truck\", \"truck\", \"trust\", \"trust\", \"trust\", \"trust\", \"trust\", \"tsang\", \"tung\", \"tv\", \"tv\", \"tv\", \"uaw\", \"uk\", \"uk\", \"uk\", \"underwrite\", \"underwrite\", \"union\", \"union\", \"union\", \"union\", \"union\", \"union\", \"user\", \"user\", \"user\", \"value\", \"value\", \"value\", \"value\", \"value\", \"value\", \"value\", \"value\", \"value\", \"vehicle\", \"vehicle\", \"vehicle\", \"vehicle\", \"vehicle\", \"vehicle\", \"vehicle\", \"visit\", \"visit\", \"visit\", \"visit\", \"wage\", \"wage\", \"walsh\", \"waste\", \"waste\", \"waste\", \"watchdog\", \"watchdog\", \"watchdog\", \"watchdog\", \"web\", \"web\", \"week\", \"week\", \"week\", \"week\", \"week\", \"week\", \"week\", \"week\", \"week\", \"week\", \"windows\", \"windows\", \"wireless\", \"wireless\", \"wmx\", \"worker\", \"worker\", \"worker\", \"worker\", \"workers\", \"workers\", \"workers\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"xiaoping\", \"xiaoping\", \"xinhua\", \"xinhua\", \"xinjiang\", \"year\", \"year\", \"year\", \"year\", \"year\", \"year\", \"year\", \"year\", \"year\", \"year\", \"yen\", \"yen\", \"yen\", \"yen\", \"yen\", \"yuan\", \"yuan\", \"yuan\", \"zemin\", \"zinc\", \"zinc\", \"zinc\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 5, 9, 3, 10, 4, 8, 7, 2, 6]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el49562867804621896767790924\", ldavis_el49562867804621896767790924_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el49562867804621896767790924\", ldavis_el49562867804621896767790924_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el49562867804621896767790924\", ldavis_el49562867804621896767790924_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x         y  topics  cluster       Freq\n",
       "topic                                                \n",
       "0     -0.069400  0.007189       1        1  17.328910\n",
       "4     -0.194081 -0.076833       2        1  16.906265\n",
       "8     -0.037245 -0.029790       3        1  14.556541\n",
       "2      0.185461 -0.015643       4        1  13.805324\n",
       "9     -0.000539 -0.006789       5        1   8.437634\n",
       "3     -0.075262 -0.067386       6        1   7.977310\n",
       "7      0.016381 -0.093135       7        1   7.187180\n",
       "6      0.030961 -0.014622       8        1   6.407350\n",
       "1      0.244465  0.041142       9        1   4.057657\n",
       "5     -0.100740  0.255867      10        1   3.335829, topic_info=            Term         Freq        Total Category  logprob  loglift\n",
       "112         bank  3158.000000  3158.000000  Default  30.0000  30.0000\n",
       "218        china  3599.000000  3599.000000  Default  29.0000  29.0000\n",
       "731         kong  2394.000000  2394.000000  Default  28.0000  28.0000\n",
       "632         hong  2379.000000  2379.000000  Default  27.0000  27.0000\n",
       "959      percent  5501.000000  5501.000000  Default  26.0000  26.0000\n",
       "...          ...          ...          ...      ...      ...      ...\n",
       "1221       share   157.697294  3333.806371  Topic10  -4.6351   0.3493\n",
       "132      billion   130.387172  3254.094879  Topic10  -4.8253   0.1833\n",
       "1010   president   107.512722   887.953634  Topic10  -5.0182   1.2891\n",
       "959      percent   109.591783  5501.563943  Topic10  -4.9991  -0.5156\n",
       "590   government   105.267364  1978.737438  Topic10  -5.0393   0.4667\n",
       "\n",
       "[572 rows x 6 columns], token_table=      Topic      Freq    Term\n",
       "term                         \n",
       "2         1  0.238596     abn\n",
       "2         2  0.079532     abn\n",
       "2         5  0.670342     abn\n",
       "5         1  0.013245  access\n",
       "5         2  0.013245  access\n",
       "...     ...       ...     ...\n",
       "1496      7  0.008159    yuan\n",
       "1498      4  0.991199   zemin\n",
       "1499      6  0.052513    zinc\n",
       "1499      7  0.052513    zinc\n",
       "1499     10  0.886154    zinc\n",
       "\n",
       "[1967 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[1, 5, 9, 3, 10, 4, 8, 7, 2, 6])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyLDAvis.sklearn.prepare(lda, tf_large, vectorizer, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning:** there is a small bug that when you show the `pyLDAvis` visualization it will hide some of the icons of JupyterLab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Neural Networks</span><a id='nn_ml'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interested? Check out the Stanford course CS224n ([Page](http://web.stanford.edu/class/cs224n/index.html#schedule))!   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "48px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}