Added EnsembleLda for stable LDA topics

docs/notebooks/ensemble_lda_with_opinosis.ipynb

@@ -0,0 +1,177 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "from gensim.models import EnsembleLda, LdaMulticore\n",
+    "from gensim.corpora import OpinosisCorpus\n",
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "enable the ensemble logger to show what it is doing currently"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda_logger = logging.getLogger(EnsembleLda.__module__)\n",
+    "elda_logger.setLevel(logging.INFO)\n",
+    "elda_logger.addHandler(logging.StreamHandler())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def pretty_print_topics():\n",
+    "    # note that the words are stemmed so they appear chopped off\n",
+    "    for t in elda.print_topics(num_words=7):\n",
+    "        print('-', t[1].replace('*',' ').replace('\"','').replace(' +',','), '\\n')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Experiments on the Opinosis Dataset\n",
+    "\n",
+    "Opinosis [1] is a small (but redundant) corpus that contains 289 product reviews for 51 products. Since it's so small, the results are rather unstable.\n",
+    "\n",
+    "[1] Kavita Ganesan, ChengXiang Zhai, and Jiawei Han, _Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions [online],_ Proceedings of the 23rd International Conference on Computational Linguistics, Association for Computational Linguistics, 2010, pp. 340–348. Available from: https://kavita-ganesan.com/opinosis/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Preparing the corpus\n",
+    "\n",
+    "First, download the opinosis dataset. On linux it can be done like this for example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!mkdir ~/opinosis\n",
+    "!wget -P ~/opinosis https://github.com/kavgan/opinosis/raw/master/OpinosisDataset1.0_0.zip\n",
+    "!unzip ~/opinosis/OpinosisDataset1.0_0.zip -d ~/opinosis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "path = os.path.expanduser('~/opinosis/')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Corpus and id2word mapping can be created using the load_opinosis_data function provided in the package.\n",
+    "It preprocesses the data using the PorterStemmer and stopwords from the nltk package.\n",
+    "\n",
+    "The parameter of the function is the relative path to the folder, into which the zip file was extracted before. That folder contains a 'summaries-gold' subfolder."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "opinosis = OpinosisCorpus(path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**parameters**\n",
+    "\n",
+    "**topic_model_kind** ldamulticore is highly recommended for EnsembleLda. ensemble_workers and **distance_workers** are used to improve the time needed to train the models, as well as the **masking_method** 'rank'. ldamulticore is not able to fully utilize all cores on this small corpus, so **ensemble_workers** can be set to 3 to get 95 - 100% cpu usage on my i5 3470.\n",
+    "\n",
+    "Since the corpus is so small, a high number of **num_models** is needed to extract stable topics. The Opinosis corpus contains 51 categories, however, some of them are quite similar. For example there are 3 categories about the batteries of portable products. There are also multiple categories about cars. So I chose 20 for num_topics, which is smaller than the number of categories."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda = EnsembleLda(\n",
+    "    corpus=opinosis.corpus, id2word=opinosis.id2word, num_models=128, num_topics=20,\n",
+    "    passes=20, iterations=100, ensemble_workers=3, distance_workers=4,\n",
+    "    topic_model_class='ldamulticore', masking_method='rank',\n",
+    ")\n",
+    "pretty_print_topics()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The default for **min_samples** would be 64, half of the number of models and **eps** would be 0.1. You basically play around with them until you find a sweetspot that fits for your needs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda.recluster(min_samples=55, eps=0.14)\n",
+    "pretty_print_topics()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/src/apiref.rst

@@ -22,13 +22,15 @@ Modules:
     corpora/malletcorpus
     corpora/mmcorpus
     corpora/_mmreader
+    corpora/opinosiscorpus
     corpora/sharded_corpus
     corpora/svmlightcorpus
     corpora/textcorpus
     corpora/ucicorpus
     corpora/wikicorpus
     models/ldamodel
     models/ldamulticore
+    models/ensemblelda
     models/nmf
     models/lsimodel
     models/ldaseqmodel

docs/src/auto_examples/howtos/run_doc.ipynb

@@ -15,14 +15,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "\nHow to Author Gensim Documentation\n==================================\n\nHow to author documentation for Gensim.\n\n"
+        "\n# How to Author Gensim Documentation\n\nHow to author documentation for Gensim.\n"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Background\n----------\n\nGensim is a large project with a wide range of functionality.\nUnfortunately, not all of this functionality is documented **well**, and some of it is not documented at all.\nWithout good documentation, users are unable to unlock Gensim's full potential.\nTherefore, authoring new documentation and improving existing documentation is of great value to the Gensim project.\n\nIf you implement new functionality in Gensim, please include **helpful** documentation.\nBy \"helpful\", we mean that your documentation answers questions that Gensim users may have.\nFor example:\n\n- What is this new functionality?\n- **Why** is it important?\n- **How** is it relevant to Gensim?\n- **What** can I do with it? What are some real-world applications?\n- **How** do I use it to achieve those things?\n- ... and others (if you can think of them, please add them here)\n\nBefore you author documentation, I suggest reading\n`\"What nobody tells you about documentation\" <https://www.divio.com/blog/documentation/>`__\nor watching its `accompanying video <https://www.youtube.com/watch?v=t4vKPhjcMZg>`__\n(or even both, if you're really keen).\n\nThe summary of the above presentation is: there are four distinct kinds of documentation, and you really need them all:\n\n1. Tutorials\n2. Howto guides\n3. Explanations\n4. References\n\nEach kind has its own intended audience, purpose, and writing style.\nWhen you make a PR with new functionality, please consider authoring each kind of documentation.\nAt the very least, you will (indirectly) author reference documentation through module, class and function docstrings.\n\nMechanisms\n----------\n\nWe keep our documentation as individual Python scripts.\nThese scripts live under :file:`docs/src/gallery` in one of several subdirectories:\n\n- core: core tutorials.  We try to keep this part small, avoid putting stuff here.\n- tutorials: tutorials.\n- howtos: howto guides.\n\nPick a subdirectory and save your script under it.\nPrefix the name of the script with ``run_``: this way, the the documentation builder will run your script each time it builds our docs.\n\nThe contents of the script are straightforward.\nAt the very top, you need a docstring describing what your script does.\n\n"
+        "## Background\n\nGensim is a large project with a wide range of functionality.\nUnfortunately, not all of this functionality is documented **well**, and some of it is not documented at all.\nWithout good documentation, users are unable to unlock Gensim's full potential.\nTherefore, authoring new documentation and improving existing documentation is of great value to the Gensim project.\n\nIf you implement new functionality in Gensim, please include **helpful** documentation.\nBy \"helpful\", we mean that your documentation answers questions that Gensim users may have.\nFor example:\n\n- What is this new functionality?\n- **Why** is it important?\n- **How** is it relevant to Gensim?\n- **What** can I do with it? What are some real-world applications?\n- **How** do I use it to achieve those things?\n- ... and others (if you can think of them, please add them here)\n\nBefore you author documentation, I suggest reading\n`\"What nobody tells you about documentation\" <https://www.divio.com/blog/documentation/>`__\nor watching its `accompanying video <https://www.youtube.com/watch?v=t4vKPhjcMZg>`__\n(or even both, if you're really keen).\n\nThe summary of the above presentation is: there are four distinct kinds of documentation, and you really need them all:\n\n1. Tutorials\n2. Howto guides\n3. Explanations\n4. References\n\nEach kind has its own intended audience, purpose, and writing style.\nWhen you make a PR with new functionality, please consider authoring each kind of documentation.\nAt the very least, you will (indirectly) author reference documentation through module, class and function docstrings.\n\n## Mechanisms\n\nWe keep our documentation as individual Python scripts.\nThese scripts live under :file:`docs/src/gallery` in one of several subdirectories:\n\n- core: core tutorials.  We try to keep this part small, avoid putting stuff here.\n- tutorials: tutorials.\n- howtos: howto guides.\n\nPick a subdirectory and save your script under it.\nPrefix the name of the script with ``run_``: this way, the the documentation builder will run your script each time it builds our docs.\n\nThe contents of the script are straightforward.\nAt the very top, you need a docstring describing what your script does.\n\n"
       ]
     },
     {
@@ -54,14 +54,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Authoring Workflow\n------------------\n\nThere are several ways to author documentation.\nThe simplest and most straightforward is to author your ``script.py`` from scratch.\nYou'll have the following cycle:\n\n1. Make changes\n2. Run ``python script.py``\n3. Check standard output, standard error and return code\n4. If everything works well, stop.\n5. Otherwise, go back to step 1).\n\nIf the above is not your cup of tea, you can also author your documentation as a Jupyter notebook.\nThis is a more flexible approach that enables you to tweak parts of the documentation and re-run them as necessary.\n\nOnce you're happy with the notebook, convert it to a script.py.\nThere's a helpful `script <https://github.com/mpenkov/gensim/blob/numfocus/docs/src/tools/to_python.py>`__ that will do it for you.\nTo use it::\n\n    python to_python.py < notebook.ipynb > script.py\n\nYou may have to touch up the resulting ``script.py``.\nMore specifically:\n\n- Update the title\n- Update the description\n- Fix any issues that the markdown-to-RST converter could not deal with\n\nOnce your script.py works, put it in a suitable subdirectory.\nPlease don't include your original Jupyter notebook in the repository - we won't be using it.\n\n"
+        "## Authoring Workflow\n\nThere are several ways to author documentation.\nThe simplest and most straightforward is to author your ``script.py`` from scratch.\nYou'll have the following cycle:\n\n1. Make changes\n2. Run ``python script.py``\n3. Check standard output, standard error and return code\n4. If everything works well, stop.\n5. Otherwise, go back to step 1).\n\nIf the above is not your cup of tea, you can also author your documentation as a Jupyter notebook.\nThis is a more flexible approach that enables you to tweak parts of the documentation and re-run them as necessary.\n\nOnce you're happy with the notebook, convert it to a script.py.\nThere's a helpful `script <https://github.com/RaRe-Technologies/gensim/blob/develop/docs/src/tools/to_python.py>`__ that will do it for you.\nTo use it::\n\n    python to_python.py < notebook.ipynb > script.py\n\nYou may have to touch up the resulting ``script.py``.\nMore specifically:\n\n- Update the title\n- Update the description\n- Fix any issues that the markdown-to-RST converter could not deal with\n\nOnce your script.py works, put it in a suitable subdirectory.\nPlease don't include your original Jupyter notebook in the repository - we won't be using it.\n\n"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Correctness\n-----------\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\nUsing data in your documentation\n--------------------------------\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API <https://radimrehurek.com/gensim/gensim_numfocus/auto_examples/010_tutorials/run_downloader_api.html>`__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\nFinalizing your contribution\n----------------------------\n\nFirst, get Sphinx Gallery to build your documentation::\n\n    make -C docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n    git add docs/src/gallery/tutorials/run_example.py\n    git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n    git add docs/src/auto_examples/howtos/sg_execution_times.rst\n    git commit -m \"enter a helpful commit message here\"\n    git push origin branchname\n\n.. Note::\n  You may be wondering what all those other files are.\n  Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n  The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n  Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n  Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, make a PR on `github <https://github.com/RaRe-Technologies/gensim>`__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the gallery alongside the rest of the example.\nAt that stage, give yourself a pat on the back: you're done!\n\n"
+        "## Correctness\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\n## Using data in your documentation\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API <https://radimrehurek.com/gensim/gensim_numfocus/auto_examples/010_tutorials/run_downloader_api.html>`__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\n## Finalizing your contribution\n\nFirst, get Sphinx Gallery to build your documentation::\n\n    make -C docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n    git add docs/src/gallery/tutorials/run_example.py\n    git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n    git add docs/src/auto_examples/howtos/sg_execution_times.rst\n    git commit -m \"enter a helpful commit message here\"\n    git push origin branchname\n\n.. Note::\n  You may be wondering what all those other files are.\n  Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n  The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n  Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n  Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, make a PR on `github <https://github.com/RaRe-Technologies/gensim>`__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the gallery alongside the rest of the example.\nAt that stage, give yourself a pat on the back: you're done!\n\n"
       ]
     }
   ],
@@ -81,7 +81,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.5"
+      "version": "3.9.1"
     }
   },
   "nbformat": 4,

docs/src/auto_examples/howtos/run_doc.py

@@ -111,7 +111,7 @@
 # This is a more flexible approach that enables you to tweak parts of the documentation and re-run them as necessary.
 #
 # Once you're happy with the notebook, convert it to a script.py.
-# There's a helpful `script <https://github.com/mpenkov/gensim/blob/numfocus/docs/src/tools/to_python.py>`__ that will do it for you.
+# There's a helpful `script <https://github.com/RaRe-Technologies/gensim/blob/develop/docs/src/tools/to_python.py>`__ that will do it for you.
 # To use it::
 #
 #     python to_python.py < notebook.ipynb > script.py

docs/src/auto_examples/howtos/run_doc.py.md5

@@ -1 +1 @@
-b3db0b66859316de13e1a36fa6181657
+512a76ce743dd12482d21784a76b60fe