Initial Sphinx revamp (#2459)

* Remove unused docs/package.json

Nowadays sphinx-autobuild offers similar functionality.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix Sphinx language config

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix circular imports

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Remove unused documentation extensions

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Try to not move files around

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix CSS paths

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix file hierarchy

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix Read the Docs configuration

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Update documentation dependencies

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Manually fix JupyterCommandGroup API docs

See gh-2453.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Install all documentation dependencies

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Restore test requirements

kedro.extras.datasets is still being tested.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Remove all autosummary members from problematic class

See gh-2453.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Temporarily disable nitpicky mode for linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Upgrade Read the Docs to Python 3.8

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix destination of linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Minor fixes to docstrings

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Minor reference fix

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Cap sphinx-autodoc-typehints version

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Restore nitpicky linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Restore treating documentation warnings as errors

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Remove redundant build-docs CI job

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Ignore .rst files generated to support docs build

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Explain ignored documentation files

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Move linkcheck to Read the Docs

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Remove old devserver Makefile target

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix databricks broken reference

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix pyspark broken reference

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Fix other broken references

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

---------

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

.circleci/continue_config.yml

@@ -354,34 +354,6 @@ jobs:
           name: Pip-compile requirements file
           command: conda activate kedro_builder; make pip-compile
 
-  build_docs:
-    executor:
-      name: docker
-      python_version: "3.7"
-    steps:
-      - setup
-      - run:
-          name: Build docs
-          command: make build-docs
-      - run:
-          name: Pip freeze including docs dependencies
-          command: pip freeze
-          when: always
-
-  docs_linkcheck:
-    executor:
-      name: docker
-      python_version: "3.8"
-    steps:
-      - setup
-      - run:
-          name: Check for broken links
-          command: make linkcheck
-      - run:
-          name: Pip freeze including docs dependencies
-          command: pip freeze
-          when: always
-
   sync:
     docker:
       # https://circleci.com/docs/2.0/circleci-images/#circleci-base-image
@@ -544,27 +516,6 @@ jobs:
 workflows:
   version: 2.1
 
-  build_docs_only:
-    when:
-      and:
-        - <<pipeline.parameters.docs_change>>
-        - not: <<pipeline.parameters.code_change>>
-        - not: <<pipeline.parameters.release_kedro>>
-        - not: <<pipeline.parameters.run_hourly>>
-        - not: <<pipeline.parameters.run_nightly>>
-    jobs:
-      - build_docs
-      - docs_linkcheck
-      - lint:
-          matrix:
-            parameters:
-              python_version: ["3.7", "3.8", "3.9", "3.10"]
-      - all_circleci_checks_succeeded:
-          requires:
-            - build_docs
-            - docs_linkcheck
-            - lint
-
   build_code_and_docs:
     when:
       and:
@@ -601,8 +552,6 @@ workflows:
           matrix:
             parameters:
               python_version: ["3.7", "3.8", "3.9", "3.10"]
-      - build_docs
-      - docs_linkcheck
       - all_circleci_checks_succeeded:
           requires:
             - e2e_tests
@@ -612,8 +561,6 @@ workflows:
             - lint
             - pip_compile
             - win_pip_compile
-            - build_docs
-            - docs_linkcheck
 
   main_updated:
     when:

.gitignore

@@ -134,6 +134,11 @@ venv.bak/
 /site
 /kedro/framework/html
 
+# Sphinx documentation
+# Additional files created by sphinx.ext.autosummary
+# Some of them are actually tracked to control the output
+/docs/source/kedro.*
+
 # mypy
 .mypy_cache/
 

.readthedocs.yml

@@ -9,20 +9,22 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.7"
+    python: "3.8"
     nodejs: "19"
   apt_packages:
     - libasound2
   jobs:
     post_create_environment:
       - npm install -g @mermaid-js/mermaid-cli
       - ./docs/kedro-datasets-docs.sh
+    pre_build:
+      - python -m sphinx -WETan -j auto -D language=en -b linkcheck -d _build/doctrees docs/source _build/linkcheck
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   builder: html
-  configuration: docs/conf.py
-  fail_on_warning: false  # Turn back on soon
+  configuration: docs/source/conf.py
+  fail_on_warning: true
 
 # Build documentation with MkDocs
 # mkdocs:

Makefile

@@ -44,9 +44,6 @@ linkcheck:
 	pip install -e ".[docs]"
 	./docs/build-docs.sh "linkcheck"
 
-devserver: build-docs
-	cd docs && npm install && npm start
-
 package: clean install
 	python setup.py sdist bdist_wheel
 

docs/build-docs.sh

@@ -7,35 +7,12 @@ set -o nounset
 
 action=$1
 
-# Move some files around. We need a separate build directory, which would
-# have all the files, build scripts would shuffle the files,
-# we don't want that happening on the actual code locally.
-# When running on ReadTheDocs, sphinx-build would run directly on the original files,
-# but we don't care about the code state there.
-rm -rf docs/build
+# Reinstall kedro-datasets locally
 rm -rf kedro/datasets
-mkdir docs/build/
-cp -r docs/_templates docs/conf.py docs/*.svg docs/*.json  docs/build/
-
 bash docs/kedro-datasets-docs.sh
 
 if [ "$action" == "linkcheck" ]; then
-  sphinx-build -c docs/ -ETan -j auto -D language=en -b linkcheck docs/build/ docs/build/html
+  sphinx-build -WETan -j auto -D language=en -b linkcheck -d docs/build/doctrees docs/source docs/build/linkcheck
 elif [ "$action" == "docs" ]; then
-  sphinx-build -c docs/ -ETa -j auto -D language=en docs/build/ docs/build/html
+  sphinx-build -WETa -j auto -D language=en -b html -d docs/build/doctrees docs/source docs/build/html
 fi
-
-# Clean up build artefacts
-rm -rf docs/build/html/_sources
-
-# Copy built HTML to temp directory, clean up build dir and replace with built docs only
-rm -rf docs/temp
-mkdir docs/temp/
-mkdir docs/temp/html
-cp -rf docs/build/html/* docs/temp/html
-
-rm -rf docs/build
-mkdir docs/build
-mkdir docs/build/html
-cp -rf docs/temp/html/* docs/build/html
-rm -rf docs/temp

docs/conf.py → docs/source/conf.py

@@ -15,9 +15,7 @@
 import importlib
 import os
 import re
-import shutil
 import sys
-from distutils.dir_util import copy_tree
 from inspect import getmembers, isclass, isfunction
 from pathlib import Path
 from typing import List, Tuple
@@ -49,12 +47,8 @@
     "sphinx.ext.napoleon",
     "sphinx_autodoc_typehints",
     "sphinx.ext.doctest",
-    "sphinx.ext.todo",
-    "sphinx.ext.coverage",
-    "sphinx.ext.mathjax",
     "sphinx.ext.ifconfig",
     "sphinx.ext.viewcode",
-    "nbsphinx",
     "sphinx_copybutton",
     "sphinxcontrib.mermaid",
     "myst_parser",
@@ -68,6 +62,7 @@
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
+html_static_path = ["_static"]
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
@@ -82,7 +77,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -305,23 +300,6 @@
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = False
 
-# -- Extension configuration -------------------------------------------------
-
-# nbsphinx_prolog = """
-# see here for prolog/epilog details:
-# https://nbsphinx.readthedocs.io/en/0.3.1/prolog-and-epilog.html
-# """
-
-nbsphinx_epilog = """
-.. note::
-
-     Found a bug, or didn't find what you were looking for? 🙏 `Please file a
-     ticket <https://github.com/kedro-org/kedro/issues/new/choose>`_
-"""
-
-# -- NBconvert kedro config -------------------------------------------------
-nbsphinx_kedro_name = "kedro"
-
 # -- Kedro specific configuration -----------------------------------------
 KEDRO_MODULES = [
     "kedro.io",
@@ -493,20 +471,6 @@ def autodoc_process_docstring(app, what, name, obj, options, lines):
     remove_arrows_in_examples(lines)
 
 
-def _prepare_build_dir(app, config):
-    """Get current working directory to the state expected
-    by the ReadTheDocs builder. Shortly, it does the same as
-    ./build-docs.sh script except not running `sphinx-build` step."""
-    build_root = Path(app.srcdir)
-    build_out = Path(app.outdir)
-    copy_tree(str(here / "source"), str(build_root))
-    copy_tree(str(build_root / "api_docs"), str(build_root))
-    shutil.rmtree(str(build_root / "api_docs"))
-    shutil.rmtree(str(build_out), ignore_errors=True)
-    copy_tree(str(build_root / "css"), str(build_out / "_static" / "css"))
-    shutil.rmtree(str(build_root / "css"))
-
-
 def env_override(default_appid):
     build_version = os.getenv("READTHEDOCS_VERSION")
 
@@ -533,7 +497,6 @@ def _add_jinja_filters(app):
 
 
 def setup(app):
-    app.connect("config-inited", _prepare_build_dir)
     app.connect("builder-inited", _add_jinja_filters)
     app.connect("autodoc-process-docstring", autodoc_process_docstring)
     app.add_css_file("css/qb1-sphinx-rtd.css")
@@ -573,4 +536,4 @@ def setup(app):
 # https://github.com/mermaidjs/mermaid.cli#linux-sandbox-issue
 mermaid_params = ["-p", here / "puppeteer-config.json", "-s", "2"]
 # https://github.com/kedro-org/kedro/issues/2451
-mermaid_version = mermaid_init_js = None
+mermaid_version = mermaid_init_js = ""

docs/source/contribution/development_for_databricks.md

@@ -7,7 +7,7 @@
 This guide describes how to efficiently develop features and fixes for Kedro on Databricks. Using this guide, you will be able to quickly test your locally modified version of Kedro on Databricks as part of a build-and-test development cycle.
 
 ```{note}
-This page is for people developing changes to Kedro that need to test them on Databricks. If you are working on a Kedro project and need more information about project-deployment, consult the [documentation for deploying Kedro projects on Databricks](../deployment/databricks.md).
+This page is for people developing changes to Kedro that need to test them on Databricks. If you are working on a Kedro project and need more information about workflows, consult the [documentation for developing a Kedro project on Databricks](../integrations/databricks_workspace.md).
 ```
 
 ## Prerequisites

docs/source/index.rst

@@ -186,30 +186,11 @@ Welcome to Kedro's documentation!
    deployment/airflow_astronomer
    deployment/dask
 
-.. toctree::
-   :maxdepth: 2
-   :caption: Databricks integration
-
-   databricks_integration/visualisation
-
-.. toctree::
-   :maxdepth: 2
-   :caption: PySpark integration
-
-   tools_integration/pyspark
-
-.. toctree::
-   :maxdepth: 2
-   :caption: FAQs
-
-   faq/faq
-   faq/architecture_overview
-   faq/kedro_principles
-
 .. toctree::
    :maxdepth: 2
    :caption: Resources
 
+   faq/faq
    resources/glossary
 
 

docs/source/integrations/databricks.md

@@ -5,5 +5,5 @@
 :maxdepth: 2
 
 databricks_workspace.md
-visualisation.md
+databricks_visualisation.md
 ```

docs/source/integrations/pyspark.md

@@ -0,0 +1,8 @@
+# PySpark integration
+
+```{toctree}
+:caption: PySpark
+:maxdepth: 2
+
+pyspark_integration.md
+```

docs/source/kedro.framework.cli.jupyter.JupyterCommandGroup.rst

@@ -0,0 +1,10 @@
+kedro.framework.cli.jupyter.JupyterCommandGroup
+===============================================
+
+.. currentmodule:: kedro.framework.cli.jupyter
+
+.. autoclass:: JupyterCommandGroup
+   :members:
+
+.. Removed all methods and properties,
+.. see https://github.com/kedro-org/kedro/issues/2453

docs/source/kedro_project_setup/starters.md

@@ -49,8 +49,8 @@ The Kedro team maintains the following starters for a range of Kedro projects:
 * [`astro-airflow-iris`](https://github.com/kedro-org/kedro-starters/tree/main/astro-airflow-iris): The [Kedro Iris dataset example project](../get_started/new_project.md) with a minimal setup for deploying the pipeline on Airflow with [Astronomer](https://www.astronomer.io/).
 * [`standalone-datacatalog`](https://github.com/kedro-org/kedro-starters/tree/main/standalone-datacatalog): A minimum setup to use the traditional [Iris dataset](https://www.kaggle.com/uciml/iris) with Kedro's [`DataCatalog`](../data/data_catalog.md), which is a core component of Kedro. This starter is of use in the exploratory phase of a project. For more information, read the guide to [standalone use of the `DataCatalog`](../notebooks_and_ipython/kedro_and_notebooks.md). This starter was formerly known as `mini-kedro`.
 * [`pandas-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pandas-iris): The [Kedro Iris dataset example project](../get_started/new_project.md)
-* [`pyspark-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris): An alternative Kedro Iris dataset example, using [PySpark](../tools_integration/pyspark.md)
-* [`pyspark`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark): The configuration and initialisation code for a [Kedro pipeline using PySpark](../tools_integration/pyspark.md)
+* [`pyspark-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris): An alternative Kedro Iris dataset example, using [PySpark](../integrations/pyspark_integration.md)
+* [`pyspark`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark): The configuration and initialisation code for a [Kedro pipeline using PySpark](../integrations/pyspark_integration.md)
 * [`spaceflights`](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights): The [spaceflights tutorial](../tutorial/spaceflights_tutorial.md) example code
 
 ## Starter versioning

docs/source/nodes_and_pipelines/run_a_pipeline.md

@@ -46,7 +46,7 @@ kedro run --runner=ThreadRunner
 `SparkDataSet` doesn't work correctly with `ParallelRunner`. To add concurrency to the pipeline with `SparkDataSet`, you must use `ThreadRunner`.
 ```
 
-For more information on how to maximise concurrency when using Kedro with PySpark, please visit our guide on [how to build a Kedro pipeline with PySpark](../tools_integration/pyspark.md).
+For more information on how to maximise concurrency when using Kedro with PySpark, please visit our guide on [how to build a Kedro pipeline with PySpark](../integrations/pyspark_integration.md).
 
 hook_manager: PluginManager = None,
 

docs/source/tutorial/add_another_pipeline.md

@@ -518,7 +518,7 @@ kedro run --runner=ThreadRunner
 kedro run --runner=module.path.to.my.runner
 ```
 
-`ParallelRunner` performs task parallelisation via multiprocessing, while `ThreadRunner` is intended for use with remote execution engines such as [Spark](../tools_integration/pyspark.md) and [Dask](/kedro.datasets.dask.ParquetDataSet).
+`ParallelRunner` performs task parallelisation via multiprocessing, while `ThreadRunner` is intended for use with remote execution engines such as [Spark](../integrations/pyspark_integration.md) and [Dask](/kedro.datasets.dask.ParquetDataSet).
 
 You can find out more about the runners Kedro provides, and how to create your own, in the [pipeline documentation about runners](../nodes_and_pipelines/run_a_pipeline.md).
 atasets to work with different data formats (including CSV, Excel, and Parquet)

docs/source/tutorial/package_a_project.md

@@ -151,4 +151,4 @@ There are various methods to deploy packaged pipelines via Kedro plugins:
 
 * [Kedro-Docker](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker) plugin for packaging and shipping Kedro projects within [Docker](https://www.docker.com/) containers.
 * [Kedro-Airflow](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-airflow) to convert your Kedro project into an [Airflow](https://airflow.apache.org/) project.
-* The [Deployment guide](../deployment/deployment_guide) touches on other deployment targets such as AWS Batch and Prefect, and there is a [range of third-party plugins for deployment](extend_kedro/plugins.md#community-developed-plugins).
+* The [Deployment guide](../deployment/deployment_guide) touches on other deployment targets such as AWS Batch and Prefect, and there is a [range of third-party plugins for deployment](../extend_kedro/plugins.md#community-developed-plugins).