Text Summarization with Sumy Python .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Summarization Using Sumy & Python\n",
    "##### Installation\n",
    "+ pip install sumy\n",
    "\n",
    "+ offers several algorithms and methods for summarization\n",
    " - Luhn – heurestic method\n",
    " - Latent Semantic Analysis\n",
    " - Edmundson heurestic method with previous statistic research\n",
    "\n",
    " - LexRank – Unsupervised approach inspired by algorithms PageRank and HITS\n",
    " - TextRank\n",
    " - SumBasic – Method that is often used as a baseline in the literature\n",
    " - KL-Sum – Method that greedily adds sentences to a summary so long as it decreases the KL Divergence. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Packages\n",
    "import sumy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumy.parsers.plaintext import PlaintextParser\n",
    "from sumy.nlp.tokenizers import Tokenizer\n",
    "from sumy.summarizers.lex_rank import LexRankSummarizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "document1 =\"\"\"Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to progressively improve their performance on a specific task. Machine learning algorithms build a mathematical model of sample data, known as \"training data\", in order to make predictions or decisions without being explicitly programmed to perform the task. Machine learning algorithms are used in the applications of email filtering, detection of network intruders, and computer vision, where it is infeasible to develop an algorithm of specific instructions for performing the task. Machine learning is closely related to computational statistics, which focuses on making predictions using computers. The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning. Data mining is a field of study within machine learning, and focuses on exploratory data analysis through unsupervised learning.In its application across business problems, machine learning is also referred to as predictive analytics.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Strings\n",
    "parser = PlaintextParser.from_string(document1,Tokenizer(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Files\n",
    "parser = PlaintextParser.from_file(file, Tokenizer(\"english\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using LexRank\n",
    "+ unsupervised approach to text summarization based on graph-based centrality scoring of sentences. \n",
    "+ The main idea is that sentences \"recommend\" other similar sentences to the reader. Thus, if one sentence is very similar to many others, it will likely be a sentence of great importance\n",
    "+ Standalone pkg pip install lexrank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using LexRank\n",
    "summarizer = LexRankSummarizer()\n",
    "#Summarize the document with 2 sentences\n",
    "summary = summarizer(parser.document, 2) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to progressively improve their performance on a specific task.\n",
      "Machine learning algorithms build a mathematical model of sample data, known as \"training data\", in order to make predictions or decisions without being explicitly programmed to perform the task.\n"
     ]
    }
   ],
   "source": [
    "for sentence in summary:\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Lung\n",
    "+ Based on frequency of most important words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumy.summarizers.luhn import LuhnSummarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "summarizer_luhn = LuhnSummarizer()\n",
    "summary_1 =summarizer_luhn(parser.document,2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Machine learning algorithms build a mathematical model of sample data, known as \"training data\", in order to make predictions or decisions without being explicitly programmed to perform the task.\n",
      "Data mining is a field of study within machine learning, and focuses on exploratory data analysis through unsupervised learning.In its application across business problems, machine learning is also referred to as predictive analytics.\n"
     ]
    }
   ],
   "source": [
    "for sentence in summary_1:\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using LSA\n",
    "+ Based on term frequency techniques with singular value decomposition to summarize texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumy.summarizers.lsa import LsaSummarizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "summarizer_lsa = LsaSummarizer()\n",
    "summary_2 =summarizer_lsa(parser.document,2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to progressively improve their performance on a specific task.\n",
      "Machine learning is closely related to computational statistics, which focuses on making predictions using computers.\n"
     ]
    }
   ],
   "source": [
    "for sentence in summary_2:\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Alternative Method using stopwords\n",
    "from sumy.nlp.stemmers import Stemmer\n",
    "from sumy.utils import get_stop_words\n",
    "summarizer_lsa2 = LsaSummarizer()\n",
    "summarizer_lsa2 = LsaSummarizer(Stemmer(\"english\"))\n",
    "summarizer_lsa2.stop_words = get_stop_words(\"english\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to progressively improve their performance on a specific task.\n",
      "The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n"
     ]
    }
   ],
   "source": [
    "for sentence in summarizer_lsa2(parser.document,2):\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Thanks\n",
    "# Jesse JCharis\n",
    "# J-Secur1ty\n",
    "# Jesus Saves@JCharisTech\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}