Merge pull request #696 from steenzout/sphinx_build_errors

sphinx build errors

.gitignore

@@ -1,18 +1,162 @@
-*.pyc
-*~
+.nicesetup
+
 client.cfg
-build
-dist
-luigi.egg-info
+
+hadoop_test.py
+minicluster.py
+mrrunner.py
+
 packages.tar
+
 test/data
-.nicesetup
-.tox
+
+Vagrantfile
+
 *.pickle
 *.rej
 *.orig
-.DS_Store
-.idea/
+
+
+# Created by https://www.gitignore.io
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.cache
+nosetests.xml
+coverage.xml
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+
+### Vim ###
+[._]*.s[a-w][a-z]
+[._]s[a-w][a-z]
+*.un~
+Session.vim
+.netrwhist
+*~
+
+
+### PyCharm ###
+# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm
+
 *.iml
-Vagrantfile
+
+## Directory-based project format:
+.idea/
+# if you remove the above rule, at least ignore the following:
+
+# User-specific stuff:
+# .idea/workspace.xml
+# .idea/tasks.xml
+# .idea/dictionaries
+
+# Sensitive or high-churn files:
+# .idea/dataSources.ids
+# .idea/dataSources.xml
+# .idea/sqlDataSources.xml
+# .idea/dynamic.xml
+# .idea/uiDesigner.xml
+
+# Gradle:
+# .idea/gradle.xml
+# .idea/libraries
+
+# Mongo Explorer plugin:
+# .idea/mongoSettings.xml
+
+## File-based project format:
+*.ipr
+*.iws
+
+## Plugin-specific files:
+
+# IntelliJ
+out/
+
+# mpeltonen/sbt-idea plugin
+.idea_modules/
+
+# JIRA plugin
+atlassian-ide-plugin.xml
+
+# Crashlytics plugin (for Android Studio and IntelliJ)
+com_crashlytics_export_strings.xml
+crashlytics.properties
+crashlytics-build.properties
+
+
+### Vagrant ###
 .vagrant/
+
+
+### OSX ###
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+
+# Thumbnails
+._*
+
+# Files that might appear on external disk
+.Spotlight-V100
+.Trashes
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk

.travis.yml

@@ -14,6 +14,7 @@ env:
     - TOX_ENV=cdh
     - TOX_ENV=hdp
     - TOX_ENV=nonhdfs
+    - TOX_ENV=docs
 
 sudo: false
 

README.rst

@@ -68,7 +68,7 @@ build up data files.
 Background
 ----------
 
-We use Luigi internally at `Spotify <http://www.spotify.com/>`_ to run
+We use Luigi internally at `Spotify <https://www.spotify.com/us/>`_ to run
 thousands of tasks every day, organized in complex dependency graphs.
 Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
 that powers all kinds of stuff including recommendations, toplists, A/B
@@ -80,7 +80,7 @@ can help programmers focus on the most important bits and leave the rest
 Conceptually, Luigi is similar to `GNU
 Make <http://www.gnu.org/software/make/>`_ where you have certain tasks
 and these tasks in turn may have dependencies on other tasks. There are
-also some similarities to `Oozie <http://incubator.apache.org/oozie/>`_
+also some similarities to `Oozie <http://oozie.apache.org/>`_
 and `Azkaban <http://data.linkedin.com/opensource/azkaban>`_. One major
 difference is that Luigi is not just built specifically for Hadoop, and
 it's easy to extend it with other kinds of tasks.
@@ -115,7 +115,7 @@ Who uses Luigi?
 
 Several companies have written blog posts or presentation about Luigi:
 
-* `Spotify <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`_
+* `Spotify : NYC Data Science <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`_
 * `Foursquare <http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897>`_
 * `Mortar Data <https://help.mortardata.com/technologies/luigi>`_
 * `Stripe <http://www.slideshare.net/PyData/python-as-part-of-a-production-machine-learning-stack-by-michael-manapat-pydata-sv-2014>`_
@@ -136,14 +136,14 @@ External links
 --------------
 
 * `Documentation <http://luigi.readthedocs.org/>`_, including the `Luigi package documentation <http://luigi.readthedocs.org/en/latest/api/luigi.html>`_ (Read the Docs)
-* `Mailing List <https://groups.google.com/forum/#!forum/luigi-user>`_ (Google Groups)
+* `Mailing List <https://groups.google.com/d/forum/luigi-user/>`_ (Google Groups)
 * `Releases <https://pypi.python.org/pypi/luigi>`_ (PyPi)
 * `Source code <https://github.com/spotify/luigi>`_ (Github)
 
 Authors
 -------
 
-Luigi was built at `Spotify <http://www.spotify.com/>`_, mainly by
+Luigi was built at `Spotify <https://www.spotify.com/us/>`_, mainly by
 `Erik Bernhardsson <https://github.com/erikbern>`_ and `Elias
 Freider <https://github.com/freider>`_, but many other people have
 contributed.

doc/api_overview.rst

@@ -130,7 +130,7 @@ support command line interaction and make sure to convert the input to
 the corresponding type (i.e. datetime.date instead of a string).
 
 Setting parameter value for other classes
------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 All parameters are also exposed on a class level on the command line interface.
 For instance, say you have classes TaskA and TaskB:

doc/central_scheduler.rst

@@ -47,11 +47,11 @@ To enable the task history,
 specify ``record_task_history = True`` in the
 ``[scheduler]`` section of ``client.cfg`` and
 specify ``db_connection`` under ``[task_history]``.
-The ``db_connection`` string is to used to configure the
-`SQLAlchemy engine<http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html>`_.
+The ``db_connection`` string is to used to configure the `SQLAlchemy engine
+<http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html>`_.
 When starting up,
-``luigid`` will create all the necessary tables using
-`create_all<http://docs.sqlalchemy.org/en/rel_0_9/core/metadata.html#sqlalchemy.schema.MetaData.create_all>`_.
+``luigid`` will create all the necessary tables using `create_all
+<http://docs.sqlalchemy.org/en/rel_0_9/core/metadata.html#sqlalchemy.schema.MetaData.create_all>`_.
 
 Example configuration::
 

doc/conf.py

@@ -74,7 +74,7 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['_build']
+exclude_patterns = ['_build', 'README.rst']
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
@@ -149,7 +149,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+#html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied

doc/configuration.rst

@@ -98,6 +98,7 @@ max-reschedules
 
 max-shown-tasks
   .. versionadded:: 1.0.20
+
   The maximum number of tasks returned in a task_list api call. This
   will restrict the number of tasks shown in any section in the
   visualiser. Small values can alleviate frozen browsers when there are
@@ -160,6 +161,7 @@ worker-ping-interval
 
 worker-timeout
   .. versionadded:: 1.0.20
+
   Number of seconds after which to kill a task which has been running
   for too long. This provides a default value for all tasks, which can
   be overridden by setting the worker-timeout property in any task. This

doc/example_top_artists.rst

@@ -253,14 +253,14 @@ Launching *http://localhost:8082* should show something like this:
 .. figure:: web_server.png
    :alt: Web server screenshot
 
-   Web server screenshot
+Web server screenshot
 Looking at the dependency graph
 for any of the tasks yields something like this:
 
 .. figure:: aggregate_artists.png
    :alt: Aggregate artists screenshot
 
-   Aggregate artists screenshot
+Aggregate artists screenshot
 In case your job crashes remotely due to any Python exception,
 Luigi will try to fetch the traceback and print it on standard output.
 You need `Mechanize <http://wwwsearch.sourceforge.net/mechanize/>`__ for it

doc/index.rst

@@ -3,7 +3,7 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-.. include:: ../README.rst
+.. include:: README.rst
 
 Table of Contents
 -----------------
@@ -30,6 +30,7 @@ API Reference
 
    luigi
    luigi.contrib
+   luigi.tools
 
 
 Indices and tables

luigi/contrib/mysqldb.py

@@ -21,15 +21,17 @@ class MySqlTarget(luigi.Target):
 
     def __init__(self, host, database, user, password, table, update_id):
         """
+        Initializes a MySqlTarget instance.
+
         :param host: MySql server address. Possibly a host:port string.
         :type host: str
-        :param database: Database name
+        :param database: database name.
         :type database: str
-        :param user: Database user
+        :param user: database user
         :type user: str
-        :param password : Password for specified user
+        :param password: password for specified user.
         :type password: str
-        :param update_id: An identifier for this data set
+        :param update_id: an identifier for this data set.
         :type update_id: str
         """
         if ':' in host:
@@ -48,7 +50,7 @@ def touch(self, connection=None):
         """
         Mark this update as complete.
 
-        Important: If the marker table doesn't exist,
+        IMPORTANT, If the marker table doesn't exist,
         the connection transaction will be aborted and the connection reset.
         Then the marker table will be created.
         """