natural_language_processing_classical_approach.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Natural Language Processing - A Classical Approach**\n",
    "\n",
    "This notebook is inspired from the following notebooks/pages:\n",
    "\n",
    "- [Auto Quiz](https://github.com/cloudxlab/ml/tree/master/projects/autoquiz)\n",
    "- [Scikit-learn documentation](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a Quiz using TextBlob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by importing the required modules/packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "import random\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following is the text we are going to use to create the quiz. This text is from wikipedia on World War 2 - https://en.wikipedia.org/wiki/World_War_II which we have assigned here to the variable ww2.\n",
    "\n",
    "However, can use any text file you like by loading it as follows:\n",
    "\n",
    "`f = open('path/filename.txt')`<br>\n",
    "`text = f.read()`\n",
    "\n",
    "Note the triple quotes, they are used for defining multi line string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "ww2 = '''\n",
    "World War II (often abbreviated to WWII or WW2), also known as the Second World War, was a global war that lasted from 1939 to 1945, although related conflicts began earlier. It involved the vast majority of the world's countries—including all of the great powers—eventually forming two opposing military alliances: the Allies and the Axis. It was the most widespread war in history, and directly involved more than 100 million people from over 30 countries. In a state of total war, the major participants threw their entire economic, industrial, and scientific capabilities behind the war effort, erasing the distinction between civilian and military resources.\n",
    "\n",
    "World War II was the deadliest conflict in human history, marked by 50 million to 85 million fatalities, most of which were civilians in the Soviet Union and China. It included massacres, the deliberate genocide of the Holocaust, strategic bombing, starvation, disease and the first use of nuclear weapons in history.[1][2][3][4]\n",
    "\n",
    "The Empire of Japan aimed to dominate Asia and the Pacific and was already at war with the Republic of China in 1937,[5] but the world war is generally said to have begun on 1 September 1939[6] with the invasion of Poland by Nazi Germany and subsequent declarations of war on Germany by France and the United Kingdom. Supplied by the Soviet Union, from late 1939 to early 1941, in a series of campaigns and treaties, Germany conquered or controlled much of continental Europe, and formed the Axis alliance with Italy and Japan. Under the Molotov–Ribbentrop Pact of August 1939, Germany and the Soviet Union partitioned and annexed territories of their European neighbours, Poland, Finland, Romania and the Baltic states. The war continued primarily between the European Axis powers and the coalition of the United Kingdom and the British Commonwealth, with campaigns including the North Africa and East Africa campaigns, the aerial Battle of Britain, the Blitz bombing campaign, and the Balkan Campaign, as well as the long-running Battle of the Atlantic. On 22 June 1941, the European Axis powers launched an invasion of the Soviet Union, opening the largest land theatre of war in history, which trapped the major part of the Axis military forces into a war of attrition. In December 1941, Japan attacked the United States and European colonies in the Pacific Ocean, and quickly conquered much of the Western Pacific.\n",
    "\n",
    "The Axis advance halted in 1942 when Japan lost the critical Battle of Midway, and Germany and Italy were defeated in North Africa and then, decisively, at Stalingrad in the Soviet Union. In 1943, with a series of German defeats on the Eastern Front, the Allied invasion of Sicily and the Allied invasion of Italy which brought about Italian surrender, and Allied victories in the Pacific, the Axis lost the initiative and undertook strategic retreat on all fronts. In 1944, the Western Allies invaded German-occupied France, while the Soviet Union regained all of its territorial losses and invaded Germany and its allies. During 1944 and 1945 the Japanese suffered major reverses in mainland Asia in South Central China and Burma, while the Allies crippled the Japanese Navy and captured key Western Pacific islands.\n",
    "\n",
    "The war in Europe concluded with an invasion of Germany by the Western Allies and the Soviet Union, culminating in the capture of Berlin by Soviet troops, the suicide of Adolf Hitler and the subsequent German unconditional surrender on 8 May 1945. Following the Potsdam Declaration by the Allies on 26 July 1945 and the refusal of Japan to surrender under its terms, the United States dropped atomic bombs on the Japanese cities of Hiroshima and Nagasaki on 6 August and 9 August respectively. With an invasion of the Japanese archipelago imminent, the possibility of additional atomic bombings and the Soviet invasion of Manchuria, Japan formally surrendered on 2 September 1945. Thus ended the war in Asia, cementing the total victory of the Allies.\n",
    "\n",
    "World War II changed the political alignment and social structure of the world. The United Nations (UN) was established to foster international co-operation and prevent future conflicts. The victorious great powers—China, France, the Soviet Union, the United Kingdom, and the United States—became the permanent members of the United Nations Security Council.[7] The Soviet Union and the United States emerged as rival superpowers, setting the stage for the Cold War, which lasted for the next 46 years. Meanwhile, the influence of European great powers waned, while the decolonisation of Africa and Asia began. Most countries whose industries had been damaged moved towards economic recovery. Political integration, especially in Europe, emerged as an effort to end pre-war enmities and to create a common identity.[8]\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we’ll convert our text to a `TextBlob` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ww2b = TextBlob(ww2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentence(\"\n",
       "World War II (often abbreviated to WWII or WW2), also known as the Second World War, was a global war that lasted from 1939 to 1945, although related conflicts began earlier.\")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ww2b.sentences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('World', 'NNP'),\n",
       " ('War', 'NNP'),\n",
       " ('II', 'NNP'),\n",
       " ('often', 'RB'),\n",
       " ('abbreviated', 'VBN'),\n",
       " ('to', 'TO'),\n",
       " ('WWII', 'NNP'),\n",
       " ('or', 'CC'),\n",
       " ('WW2', 'NNP'),\n",
       " ('also', 'RB'),\n",
       " ('known', 'VBN'),\n",
       " ('as', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('Second', 'NNP'),\n",
       " ('World', 'NNP'),\n",
       " ('War', 'NNP'),\n",
       " ('was', 'VBD'),\n",
       " ('a', 'DT'),\n",
       " ('global', 'JJ'),\n",
       " ('war', 'NN'),\n",
       " ('that', 'WDT'),\n",
       " ('lasted', 'VBD'),\n",
       " ('from', 'IN'),\n",
       " ('1939', 'CD'),\n",
       " ('to', 'TO'),\n",
       " ('1945', 'CD'),\n",
       " ('although', 'IN'),\n",
       " ('related', 'JJ'),\n",
       " ('conflicts', 'NNS'),\n",
       " ('began', 'VBD'),\n",
       " ('earlier', 'RBR')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ww2b.sentences[0].tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to apply different methods on our text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Go Back to the Slides**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to prepare the dictionary of parts-of-speech as the key and value is a list of words:\n",
    "\n",
    "`{part-of-speech: [word1, word2]}`\n",
    "\n",
    "We are basically grouping the words based on the parts-of-speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sposs = {}\n",
    "\n",
    "for sentence in ww2b.sentences:   \n",
    "    poss = {}\n",
    "    sposs[sentence.string] = poss;\n",
    "    for t in sentence.tags:\n",
    "        tag = t[1]\n",
    "        if tag not in poss:\n",
    "            poss[tag] = []\n",
    "        poss[tag].append(t[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'NNP': ['World', 'War', 'II', 'WWII', 'WW2', 'Second', 'World', 'War'],\n",
       " 'RB': ['often', 'also'],\n",
       " 'VBN': ['abbreviated', 'known'],\n",
       " 'TO': ['to', 'to'],\n",
       " 'CC': ['or'],\n",
       " 'IN': ['as', 'from', 'although'],\n",
       " 'DT': ['the', 'a'],\n",
       " 'VBD': ['was', 'lasted', 'began'],\n",
       " 'JJ': ['global', 'related'],\n",
       " 'NN': ['war'],\n",
       " 'WDT': ['that'],\n",
       " 'CD': ['1939', '1945'],\n",
       " 'NNS': ['conflicts'],\n",
       " 'RBR': ['earlier']}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sposs['\\nWorld War II (often abbreviated to WWII or WW2), also known as the Second World War, was a global war that lasted from 1939 to 1945, although related conflicts began earlier.']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create the blank in string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replaceIC(word, sentence):\n",
    "    insensitive_hippo = re.compile(re.escape(word), re.IGNORECASE)\n",
    "    return insensitive_hippo.sub('__________________', sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is __________________'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replaceIC(\"hippo\", \"this is hippo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `removeWord` function tries to create a blank space in a sentence. It does that by first trying to randomly selection proper-noun, and if the proper noun is not found, it selects a noun randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def removeWord(sentence, poss):\n",
    "    words = None\n",
    "    if 'NNP' in poss:\n",
    "        words = poss['NNP']\n",
    "    elif 'NN' in poss:\n",
    "        words = poss['NN']\n",
    "    else:\n",
    "        print(\"NN and NNP not found\")\n",
    "        return (None, sentence, None)\n",
    "    if len(words) > 0:\n",
    "        word = random.choice(words)\n",
    "        replaced = replaceIC(word, sentence)\n",
    "        return (word, sentence, replaced)\n",
    "    else:\n",
    "        print(\"words are empty\")\n",
    "        return (None, sentence, None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will iterate over the sentences and create blank spaces using the `removeWord` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "World War __________________ (often abbreviated to WW__________________ or WW2), also known as the Second World War, was a global war that lasted from 1939 to 1945, although related conflicts began earlier.\n",
      "\n",
      "===============\n",
      "II\n",
      "===============\n",
      "\n",
      "\n",
      "It involved the vast majority of the world's countries—including all of the great powers—eventually forming two opposing military alliances: the Allies and the __________________.\n",
      "\n",
      "===============\n",
      "Axis\n",
      "===============\n",
      "\n",
      "\n",
      "It was the most widespread __________________ in history, and directly involved more than 100 million people from over 30 countries.\n",
      "\n",
      "===============\n",
      "war\n",
      "===============\n",
      "\n",
      "\n",
      "In a state of total __________________, the major participants threw their entire economic, industrial, and scientific capabilities behind the __________________ effort, erasing the distinction between civilian and military resources.\n",
      "\n",
      "===============\n",
      "war\n",
      "===============\n",
      "\n",
      "\n",
      "World War II was the deadliest conflict in human history, marked by 50 million to 85 million fatalities, most of which were civilians in the Soviet __________________ and China.\n",
      "\n",
      "===============\n",
      "Union\n",
      "===============\n",
      "\n",
      "\n",
      "It included massacres, the deliberate genocide of the __________________, strategic bombing, starvation, disease and the first use of nuclear weapons in history.\n",
      "\n",
      "===============\n",
      "Holocaust\n",
      "===============\n",
      "\n",
      "\n",
      "[1][2][3][4]\n",
      "\n",
      "The Empire of Japan aimed to dominate Asia and the Pacific and was already at war with the Republic of China in 1937,[5] but the world war is generally said to have begun on 1 September 1939[6] with the invasion of Poland by __________________ Germany and subsequent declarations of war on Germany by France and the United Kingdom.\n",
      "\n",
      "===============\n",
      "Nazi\n",
      "===============\n",
      "\n",
      "\n",
      "Supplied by the Soviet Union, from late 1939 to early 1941, in a series of campaigns and treaties, Germany conquered or controlled much of continental Europe, and formed the Axis alliance with Italy and __________________.\n",
      "\n",
      "===============\n",
      "Japan\n",
      "===============\n",
      "\n",
      "\n",
      "Under the Molotov–Ribbentrop Pact of August 1939, Germany and the Soviet Union partitioned and annexed territories of their European neighbours, __________________, Finland, Romania and the Baltic states.\n",
      "\n",
      "===============\n",
      "Poland\n",
      "===============\n",
      "\n",
      "\n",
      "The war continued primarily between the European Axis powers and the coalition of the United Kingdom and the British __________________, with campaigns including the North Africa and East Africa campaigns, the aerial Battle of Britain, the Blitz bombing campaign, and the Balkan Campaign, as well as the long-running Battle of the Atlantic.\n",
      "\n",
      "===============\n",
      "Commonwealth\n",
      "===============\n",
      "\n",
      "\n",
      "On 22 __________________ 1941, the European Axis powers launched an invasion of the Soviet Union, opening the largest land theatre of war in history, which trapped the major part of the Axis military forces into a war of attrition.\n",
      "\n",
      "===============\n",
      "June\n",
      "===============\n",
      "\n",
      "\n",
      "In December 1941, Japan attacked the United States and European colonies in the Pacific Ocean, and quickly conquered much of the __________________ Pacific.\n",
      "\n",
      "===============\n",
      "Western\n",
      "===============\n",
      "\n",
      "\n",
      "The Axis advance halted in 1942 when __________________ lost the critical Battle of Midway, and Germany and Italy were defeated in North Africa and then, decisively, at Stalingrad in the Soviet Union.\n",
      "\n",
      "===============\n",
      "Japan\n",
      "===============\n",
      "\n",
      "\n",
      "In 1943, with a series of German defeats on the Eastern Front, the Allied invasion of Sicily and the Allied invasion of Italy which brought about Italian surrender, and Allied victories in the Pacific, the __________________ lost the initiative and undertook strategic retreat on all fronts.\n",
      "\n",
      "===============\n",
      "Axis\n",
      "===============\n",
      "\n",
      "\n",
      "In 1944, the Western Allies invaded German-occupied __________________, while the Soviet Union regained all of its territorial losses and invaded Germany and its allies.\n",
      "\n",
      "===============\n",
      "France\n",
      "===============\n",
      "\n",
      "\n",
      "During 1944 and 1945 the Japanese suffered major reverses in mainland Asia in South Central China and Burma, while the Allies crippled the Japanese __________________ and captured key Western Pacific islands.\n",
      "\n",
      "===============\n",
      "Navy\n",
      "===============\n",
      "\n",
      "\n",
      "The war in Europe concluded with an invasion of Germany by the Western Allies and the Soviet Union, culminating in the capture of Berlin by Soviet troops, the suicide of Adolf __________________ and the subsequent German unconditional surrender on 8 May 1945.\n",
      "\n",
      "===============\n",
      "Hitler\n",
      "===============\n",
      "\n",
      "\n",
      "Following the Potsdam Declaration by the Allies on 26 July 1945 and the refusal of Japan to surrender under its terms, the __________________ States dropped atomic bombs on the Japanese cities of Hiroshima and Nagasaki on 6 August and 9 August respectively.\n",
      "\n",
      "===============\n",
      "United\n",
      "===============\n",
      "\n",
      "\n",
      "With an invasion of the __________________ese archipelago imminent, the possibility of additional atomic bombings and the Soviet invasion of Manchuria, __________________ formally surrendered on 2 September 1945.\n",
      "\n",
      "===============\n",
      "Japan\n",
      "===============\n",
      "\n",
      "\n",
      "Thus ended the war in __________________, cementing the total victory of the Allies.\n",
      "\n",
      "===============\n",
      "Asia\n",
      "===============\n",
      "\n",
      "\n",
      "World War __________________ changed the political alignment and social structure of the world.\n",
      "\n",
      "===============\n",
      "II\n",
      "===============\n",
      "\n",
      "\n",
      "The __________________ited Nations (__________________) was established to foster international co-operation and prevent future conflicts.\n",
      "\n",
      "===============\n",
      "UN\n",
      "===============\n",
      "\n",
      "\n",
      "The victorious great powers—China, France, the Soviet Union, the __________________ Kingdom, and the __________________ States—became the permanent members of the __________________ Nations Security Council.\n",
      "\n",
      "===============\n",
      "United\n",
      "===============\n",
      "\n",
      "\n",
      "[7] The Soviet Union and the __________________ States emerged as rival superpowers, setting the stage for the Cold War, which lasted for the next 46 years.\n",
      "\n",
      "===============\n",
      "United\n",
      "===============\n",
      "\n",
      "\n",
      "Meanwhile, the influence of European great powers waned, while the decolonisation of Africa and __________________ began.\n",
      "\n",
      "===============\n",
      "Asia\n",
      "===============\n",
      "\n",
      "\n",
      "Most countries whose industries had been damaged moved towards economic __________________.\n",
      "\n",
      "===============\n",
      "recovery\n",
      "===============\n",
      "\n",
      "\n",
      "Political integration, especially in __________________, emerged as an effort to end pre-war enmities and to create a common identity.\n",
      "\n",
      "===============\n",
      "Europe\n",
      "===============\n",
      "\n",
      "\n",
      "NN and NNP not found\n",
      "Founded none for \n",
      "[8]\n"
     ]
    }
   ],
   "source": [
    "for sentence in sposs.keys():\n",
    "    poss = sposs[sentence]\n",
    "    (word, osentence, replaced) = removeWord(sentence, poss)\n",
    "    if replaced is None:\n",
    "        print (\"Founded none for \")\n",
    "        print(sentence)\n",
    "    else:\n",
    "        print(replaced)\n",
    "        print (\"\\n===============\")\n",
    "        print(word)\n",
    "        print (\"===============\")\n",
    "        print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Go Back to the Slides**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find Related Posts using Scikit-Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by impoting `CountVectorizer` module, and initializing it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `min_df` parameter determines how CountVectorizer treats seldom words:\n",
    "\n",
    "- If it is set to an integer, all words occurring less than that value will be dropped\n",
    "- If it is a fraction, all words that occur in less than that fraction of the overall dataset will be dropped\n",
    "\n",
    "The `max_df` parameter works in a similar manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "content = [\"How to format my hard disk\", \" Hard disk format problems \"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['disk', 'format', 'hard', 'how', 'my', 'problems', 'to']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = vectorizer.fit_transform(content)\n",
    "vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1, 1, 1, 0, 1],\n",
       "       [1, 1, 1, 0, 0, 1, 0]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 1]\n",
      " [1 1]\n",
      " [1 1]\n",
      " [1 0]\n",
      " [1 0]\n",
      " [0 1]\n",
      " [1 0]]\n"
     ]
    }
   ],
   "source": [
    "print(X.toarray().transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that the first sentence contains all the words except \"problems\", while the second contains all but \"how\", \"my\", and \"to\". In fact, these are exactly the same columns as we have seen in the preceding table. From X, we can extract a feature vector that we will use to compare two documents with each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will apply this on a toy dataset. Let's consider 5 toy posts and find the similarity with a given post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "post1 = \"This is a toy post about machine learning. Actually, it contains not much interesting stuff.\"\n",
    "post2 = \"Imaging databases can get huge.\"\n",
    "post3 = \"Most imaging databases save images permanently.\"\n",
    "post4 = \"Imaging databases store images.\"\n",
    "post5 = \"Imaging databases store images. Imaging databases store images. Imaging databases store images.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create out **vectorizer**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#samples: 5, #features: 24\n"
     ]
    }
   ],
   "source": [
    "posts = [post1, post2, post3, post4, post5]\n",
    "\n",
    "X_train = vectorizer.fit_transform(posts)\n",
    "num_samples, num_features = X_train.shape\n",
    "\n",
    "print(\"#samples: %d, #features: %d\" % (num_samples, num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we provided 5 different posts and there are 24 different words in them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1,\n",
       "        1, 1],\n",
       "       [0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0],\n",
       "       [0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0,\n",
       "        0, 0],\n",
       "       [0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "        0, 0],\n",
       "       [0, 0, 0, 0, 3, 0, 0, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0,\n",
       "        0, 0]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['about',\n",
       " 'actually',\n",
       " 'can',\n",
       " 'contains',\n",
       " 'databases',\n",
       " 'get',\n",
       " 'huge',\n",
       " 'images',\n",
       " 'imaging',\n",
       " 'interesting',\n",
       " 'is',\n",
       " 'it',\n",
       " 'learning',\n",
       " 'machine',\n",
       " 'most',\n",
       " 'much',\n",
       " 'not',\n",
       " 'permanently',\n",
       " 'post',\n",
       " 'save',\n",
       " 'store',\n",
       " 'stuff',\n",
       " 'this',\n",
       " 'toy']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n"
     ]
    }
   ],
   "source": [
    "print(len(vectorizer.get_feature_names()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_post = \"imaging databases \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_post_vec = vectorizer.transform([new_post])\n",
    "new_post_vec.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we find the distance between the two vectors. We can measure this distance using the **Euclidean Distance**. But first we will normalize each vectors. The `scipy.linalg` module provides a function called norm. The `norm()` function calculates the Euclidean norm (shortest distance). So this will first normalize the vectors and then find the distance between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dist_norm(v1, v2):\n",
    "    v1_normalized = v1/sp.linalg.norm(v1.toarray())\n",
    "    v2_normalized = v2/sp.linalg.norm(v2.toarray())\n",
    "    delta = v1_normalized - v2_normalized\n",
    "    return sp.linalg.norm(delta.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Post 0 with dist=1.41: This is a toy post about machine learning. Actually, it contains not much interesting stuff.\n",
      "=== Post 1 with dist=0.86: Imaging databases can get huge.\n",
      "=== Post 2 with dist=0.92: Most imaging databases save images permanently.\n",
      "=== Post 3 with dist=0.77: Imaging databases store images.\n",
      "=== Post 4 with dist=0.77: Imaging databases store images. Imaging databases store images. Imaging databases store images.\n",
      "Best post is 3 with dist=0.77\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import scipy as sp\n",
    "best_doc = None\n",
    "best_dist = sys.maxsize\n",
    "best_i = None\n",
    "for i, post in enumerate(posts):\n",
    "    post_vec = X_train.getrow(i)\n",
    "    d = dist_norm(post_vec, new_post_vec)\n",
    "    print(\"=== Post %i with dist=%.2f: %s\"%(i, d, post))\n",
    "    if d<best_dist:\n",
    "        best_dist = d\n",
    "        best_i = i\n",
    "print(\"Best post is %i with dist=%.2f\"%(best_i, best_dist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Go Back to the Slides**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the file contents and the categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is called \"Twenty Newsgroups\". Here is their [official website](http://qwone.com/~jason/20Newsgroups/).\n",
    "\n",
    "In the following we will use the built-in dataset loader for 20 newsgroups from scikit-learn. In order to get faster execution times for this first example we will work on a partial dataset with only 4 categories out of the 20 available in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load the list of files matching those categories as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "twenty_train = fetch_20newsgroups(data_home='/cxldata/scikit_learn_data', subset='train', categories=categories, shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned dataset is a `scikit-learn` \"bunch\": a simple holder object with fields that can be both accessed as python dict keys or object attributes for convenience, for instance the `target_names` holds the list of the requested category names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The files themselves are loaded in memory in the data attribute. For reference the filenames are also available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(twenty_train.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(twenty_train.filenames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s print the first lines of the first loaded file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: sd345@city.ac.uk (Michael Collier)\n",
      "Subject: Converting images to HP LaserJet III?\n",
      "Nntp-Posting-Host: hampton\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(twenty_train.data[0].split(\"\\n\")[:3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "comp.graphics\n"
     ]
    }
   ],
   "source": [
    "print(twenty_train.target_names[twenty_train.target[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For speed and space efficiency reasons `scikit-learn` loads the target attribute as an array of integers that corresponds to the index of the category name in the `target_names` list. The category integer id of each sample is stored in the `target` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 3, 3, 3, 3, 3, 2, 2, 2])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to get back the category names as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "comp.graphics\n",
      "comp.graphics\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "sci.med\n",
      "sci.med\n",
      "sci.med\n"
     ]
    }
   ],
   "source": [
    "for t in twenty_train.target[:10]:\n",
    "    print(twenty_train.target_names[t])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract feature vectors suitable for machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will apply the bag of word approach. Tokenizing text with scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35788)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer()\n",
    "X_train_counts = count_vect.fit_transform(twenty_train.data)\n",
    "X_train_counts.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CountVectorizer` supports counts of **N-grams** of words or consecutive characters. Once fitted, the vectorizer has built a dictionary of feature indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4690"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.vocabulary_['algorithm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32142"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.vocabulary_[u'the']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index value of a word in the vocabulary is linked to its frequency in the whole training corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Go Back to the Slides**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a linear model to perform categorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35788)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)\n",
    "X_train_tf = tf_transformer.transform(X_train_counts)\n",
    "X_train_tf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "TfidfTransformer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35788)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_transformer = TfidfTransformer()\n",
    "X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
    "X_train_tfidf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 32270)\t0.18781918880023565\n",
      "  (0, 18474)\t0.1570845575454142\n",
      "  (0, 10071)\t0.9695609285130992\n",
      "  (1, 32139)\t0.1529256887887155\n",
      "  (1, 18474)\t0.1447113185933946\n",
      "  (1, 15515)\t0.977584967140687\n"
     ]
    }
   ],
   "source": [
    "result = tfidf_transformer.transform(count_vect.transform(['This is a cow', 'That is a goat']))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this 32270\n",
      "is 18474\n",
      "Skipping  a\n",
      "cow 10071\n"
     ]
    }
   ],
   "source": [
    "for s in 'this is a cow'.split():\n",
    "    if s in count_vect.vocabulary_:\n",
    "        print(s, count_vect.vocabulary_[s])\n",
    "    else:\n",
    "        print('Skipping ', s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll start with a naïve Bayes classifier, which provides a nice baseline for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The multinomial variant of Naive Bayes is one the most suitable for word counts tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To try to predict the outcome on a new document we need to extract the features using almost the same feature extracting chain as before. The difference is that we call `transform` instead of `fit_transform` on the transformers, since they have already been fit to the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_new = ['God is love', 'OpenGL on the GPU is fast']\n",
    "X_new_counts = count_vect.transform(docs_new)\n",
    "X_new_tfidf = tfidf_transformer.transform(X_new_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = clf.predict(X_new_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'God is love' => soc.religion.christian\n",
      "'OpenGL on the GPU is fast' => comp.graphics\n"
     ]
    }
   ],
   "source": [
    "for doc, category in zip(docs_new, predicted):\n",
    "    print('%r => %s' % (doc, twenty_train.target_names[category]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to make the vectorizer => transformer => classifier easier to work with, `scikit-learn` provides a `Pipeline` class that behaves like a compound classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "text_clfnb = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', MultinomialNB()),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names `vect`, `tfidf` and `clf` (classifier) are arbitrary. We will use them to perform grid search for suitable hyperparameters below. We can now train the model with a single command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=None)),\n",
       "                ('tfidf',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('clf',\n",
       "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clfnb.fit(twenty_train.data, twenty_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us perform performance evaluation on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8348868175765646"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "twenty_test = fetch_20newsgroups(data_home='/cxldata/scikit_learn_data', subset='test', categories=categories, shuffle=True, random_state=42)\n",
    "docs_test = twenty_test.data\n",
    "predictednb = text_clfnb.predict(docs_test)\n",
    "np.mean(predictednb == twenty_test.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We achieved 83.488% accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        precision    recall  f1-score   support\n",
      "\n",
      "           alt.atheism       0.97      0.60      0.74       319\n",
      "         comp.graphics       0.96      0.89      0.92       389\n",
      "               sci.med       0.97      0.81      0.88       396\n",
      "soc.religion.christian       0.65      0.99      0.78       398\n",
      "\n",
      "              accuracy                           0.83      1502\n",
      "             macro avg       0.89      0.82      0.83      1502\n",
      "          weighted avg       0.88      0.83      0.84      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(metrics.classification_report(twenty_test.target, predictednb, target_names=twenty_test.target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use a grid search strategy to find a good configuration of both the feature extraction components and the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we’ll use a different classifier and compute the performance metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "text_clf = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=42, max_iter=5, tol=None)),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=Non...\n",
       "                ('clf',\n",
       "                 SGDClassifier(alpha=0.001, average=False, class_weight=None,\n",
       "                               early_stopping=False, epsilon=0.1, eta0=0.0,\n",
       "                               fit_intercept=True, l1_ratio=0.15,\n",
       "                               learning_rate='optimal', loss='hinge',\n",
       "                               max_iter=5, n_iter_no_change=5, n_jobs=None,\n",
       "                               penalty='l2', power_t=0.5, random_state=42,\n",
       "                               shuffle=True, tol=None, validation_fraction=0.1,\n",
       "                               verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf.fit(twenty_train.data, twenty_train.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9101198402130493"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted = text_clf.predict(docs_test)\n",
    "np.mean(predicted == twenty_test.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We achieved 91% accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        precision    recall  f1-score   support\n",
      "\n",
      "           alt.atheism       0.95      0.80      0.87       319\n",
      "         comp.graphics       0.87      0.98      0.92       389\n",
      "               sci.med       0.94      0.89      0.91       396\n",
      "soc.religion.christian       0.90      0.95      0.93       398\n",
      "\n",
      "              accuracy                           0.91      1502\n",
      "             macro avg       0.91      0.91      0.91      1502\n",
      "          weighted avg       0.91      0.91      0.91      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "print(metrics.classification_report(twenty_test.target, predicted, target_names=twenty_test.target_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        precision    recall  f1-score   support\n",
      "\n",
      "           alt.atheism       0.97      0.60      0.74       319\n",
      "         comp.graphics       0.96      0.89      0.92       389\n",
      "               sci.med       0.97      0.81      0.88       396\n",
      "soc.religion.christian       0.65      0.99      0.78       398\n",
      "\n",
      "              accuracy                           0.83      1502\n",
      "             macro avg       0.89      0.82      0.83      1502\n",
      "          weighted avg       0.88      0.83      0.84      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(metrics.classification_report(twenty_test.target, predictednb, target_names=twenty_test.target_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[256,  11,  16,  36],\n",
       "       [  4, 380,   3,   2],\n",
       "       [  5,  35, 353,   3],\n",
       "       [  5,  11,   4, 378]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.confusion_matrix(twenty_test.target, predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected the confusion matrix shows that posts from the newsgroups on atheism and Christianity are more often confused for one another than with computer graphics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Parameter tuning using grid search**\n",
    "\n",
    "Since there are different parameters which we can choose, we’ll apply grid search to find the best parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "parameters = {\n",
    "    'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "    'tfidf__use_idf': (True, False),\n",
    "    'clf__alpha': (1e-2, 1e-3),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we’ll be applying grid search for the parameters - `ngram_range`, `use_idf` and `alpha`. If we have multiple CPU cores at our disposal, we can tell the grid searcher to try these eight parameter combinations in parallel with the `n_jobs` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs_clf = GridSearchCV(text_clf, parameters, cv=5, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs_clf = gs_clf.fit(twenty_train.data[:400], twenty_train.target[:400])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will predict and find the best score. The result of calling `fit` on a `GridSearchCV` object is a classifier that we can use to `predict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'soc.religion.christian'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target_names[gs_clf.predict(['God is love'])[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object’s `best_score_` and `best_params_` attributes store the best mean score and the parameters setting corresponding to that score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9175000000000001"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clf__alpha: 0.001\n",
      "tfidf__use_idf: True\n",
      "vect__ngram_range: (1, 1)\n"
     ]
    }
   ],
   "source": [
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"%s: %r\" % (param_name, gs_clf.best_params_[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Go Back to the Slides**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8135818908122503\n",
      "                        precision    recall  f1-score   support\n",
      "\n",
      "           alt.atheism       0.94      0.64      0.76       319\n",
      "         comp.graphics       0.71      0.95      0.82       389\n",
      "               sci.med       0.90      0.69      0.78       396\n",
      "soc.religion.christian       0.81      0.93      0.87       398\n",
      "\n",
      "              accuracy                           0.81      1502\n",
      "             macro avg       0.84      0.81      0.81      1502\n",
      "          weighted avg       0.84      0.81      0.81      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "text_clf = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', RandomForestClassifier(n_estimators=200, min_samples_split=4)),\n",
    "])\n",
    "\n",
    "text_clf.fit(twenty_train.data, twenty_train.target)\n",
    "\n",
    "predicted = text_clf.predict(docs_test)\n",
    "print(np.mean(predicted == twenty_test.target))\n",
    "\n",
    "from sklearn import metrics\n",
    "\n",
    "print(metrics.classification_report(twenty_test.target, predicted, target_names=twenty_test.target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Apply The Deep Neural networks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}