Initial Sphinx revamp

[astrojuanlu]

Description

First round at addressing gh-2454.

Development notes

The most important change of the PR is that there was some magic in build-docs.sh and conf.py that was shuffling files around which made reasoning about the docs process very difficult. I don't know the purpose of this shuffling, but probably it was to "hide" some of the files generated by sphinx.ext.autosummary from polluting the version control. The downside of this change is that, after building the docs, the checkout contains lots of untracked files. But they can be safely ignored, and a git clean -f docs/source will wipe them.

The upsides are:

• We now follow more "standard" Sphinx terminology, rather than abusing the build/ directory as source directory
• The build-docs.sh script is basically the kedro-datasets wrangling + a standard sphinx-build call
• The conf.py has much less sorcery in it

Other changes:

• Remove unused docs/package.json (if interested, use https://pypi.org/project/sphinx-autobuild/ instead)
• Remove unused Sphinx extensions (to my knowledge we are not using notebooks as docs yet, we can reintroduce later - cc @AntonyMilneQB)
• Fix some Sphinx configurations that break in newer versions (language was invalid) as well as some minor things (some mermaid configs I added with the wrong types)
• Fix some circular imports that were some static analysis tools were failing to resolve (although said tools were not introduced in this PR)
• Upgrade to more modern Sphinx version and dependencies (not to the latest Sphinx version yet, because that requires adjustments to the CSS due to some changes in docutils 0.17 onwards)
• More streamlined process to install all the documentation dependencies
• Remove redundant build_docs Circle CI job (Read the Docs takes care of it for now)
• Strict modes are back on

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

linkcheck is failing, I'll have a look at that later.

[astrojuanlu]

Rendered docs 🤖 https://kedro--2459.org.readthedocs.build/en/2459/

[datajoely]

Non trivial change @astrojuanlu - thank you!

[astrojuanlu]

The linkcheck errors appear to be related to the docstrings of inherited members, and are present in current main, so they were not introduced in this PR: gh-2453.

[stichbury]

Not a straightforward task you've taken on there -- thank you!

I've pulled down your branch and removed the build/html docs so I'm sure everything is clean. I've run make build-docs locally but am still seeing the same 70 errors I usually get on a local build. Should I do some kind of pip install to get Kedro dependencies into my virtual environment first? I've attached the output.txt

output.txt

Working on Mac M1, Python 3.8. I've attached the output of conda list so you can see what I've got in my env.

condalisting.txt

[astrojuanlu]

@stichbury You should be able to do pip install -e ".[docs]" with this branch and it should bring all the dependencies now. These are the steps I followed:

> micromamba create -n kedro38-dev python=3.8 hdf5 gdal "numpy=1.22.4" -c conda-forge -y  # conda also works
> micromamba activate kedro38-dev
(kedro38-dev) > pip install -e ".[docs]"

[astrojuanlu]

I'm chasing down some broken references in certain API pages, but again I am not able to reproduce locally what I see on Read the Docs, regardless of this PR, which is making any debugging very difficult.

For instance, https://docs.kedro.org/en/0.18.7/kedro.pipeline.Pipeline.html shows links in the autosummary tables:

<a class="reference internal" href="#kedro.pipeline.Pipeline.grouped_nodes" title="kedro.pipeline.Pipeline.grouped_nodes"><code class="xref py py-obj docutils literal notranslate"><span class="pre">grouped_nodes</span></code></a>

But locally, after checking out 0.18.7 and with the same versions of the dependencies, none of those references work:

<code class="xref py py-obj docutils literal notranslate"><span class="pre">grouped_nodes</span></code>

The next step is to try to reproduce exactly what RTD is doing, for example by looking at the 0.18.7 build: https://readthedocs.org/projects/kedro/builds/19869393/

[astrojuanlu]

Final warnings to address:

/home/circleci/project/docs/source/kedro.io.DataCatalog.rst:20: WARNING: Field list ends without a blank line; unexpected unindent.
/home/circleci/project/docs/source/kedro.io.DataCatalog.rst:17: WARNING: Field list ends without a blank line; unexpected unindent.
/home/circleci/project/kedro/io/data_catalog.py:docstring of kedro.io.data_catalog.DataCatalog:7: WARNING: Field list ends without a blank line; unexpected unindent.
/home/circleci/project/docs/source/tutorial/package_a_project.md:154: WARNING: 'myst' cross-reference target not found: 'extend_kedro/plugins.md#community-developed-plugins' [myst.xref_missing]

[antonymilne]

This is heroic work 😮 💯

In response to something you mentioned above:

The trick is to do pip install -r test_requirements.txt, which essentially duplicates the dependencies of pip install kedro-datasets[all] (why??)

Historical reasons I believe. All datasets used to be (and still are) defined in kedro.extras. To test these in CI, long ago someone put in the step pip install -r test_requirements.txt to get all their requirements.

The introduction of extra_requires so that a user can do kedro[pandas.CSVDataSet] etc. is relatively recent (like 2+ years ago I think, but I'm old so it still feels recent) and has been copied over to the same thing in kedro-datasets. test_requirements.txt should be redundant now since you can use the extra_requires instead, just no one has changed it.

Please do let me know if you figure out any way we can get rid of those files like docs/source/kedro.config.rst etc. From memory, I think the reason I needed to keep them was because if sphinx tries to automatically document them, it follows through to packages outside kedro, which triggers all sorts of errors 😬 Here's the gory details in case you're interested: https://github.com/quantumblacklabs/private-kedro/pull/1094 and https://github.com/quantumblacklabs/private-kedro/pull/1094#issuecomment-838792714 (coming from someone who doesn't know sphinx very well but spent a long time battling against it at the time)

[astrojuanlu]

Thanks a lot for the extra context @AntonyMilneQB! I've had some painful debugging sessions with Sphinx over the years but I don't remember anything like this...

For the record, I was considering sphinx-autoapi or the newer https://sphinx-autodoc2.readthedocs.io/ to replace sphinx.ext.autosummary, since those do not need to import the code (and hence would save us from installing all the datasets dependencies just to build their docs). That will go in a separate PR though.

Please do let me know if you figure out any way we can get rid of those files like docs/source/kedro.config.rst etc. From memory, I think the reason I needed to keep them was because if sphinx tries to automatically document them, it follows through to packages outside kedro, which triggers all sorts of errors 😬

Yeah, that's what I observed as well. What I understand is that all tools will struggle with documenting classes that are available in two places - there's not much we can do about it, only:

• Keep a modified version of some of the automatically generated files for better control (what we're doing now), or
• Make some of the modules private (similar to what I'm advocating in Narrow down kedro-datasets public API kedro-plugins#139)

In any case, ✨ this PR is done ✨ and only needs an extra nudge on the flaky Windows tests for all the checks to be green. cc @stichbury

[stichbury]

@astrojuanlu I'm trying to build locally but am hitting some issues that I didn't have previously. I resolved a complaint about HDF5 but this one for Fiona is not so straightforward:

Collecting fiona>=1.8
  Using cached Fiona-1.9.2.tar.gz (919 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      <string>:82: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
      WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
      CRITICAL:root:A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make: *** [build-docs] Error 1

[astrojuanlu]

@stichbury The [Errno 2] No such file or directory: 'gdal-config' tells me that GDAL is not installed - it's included in the command I gave above, could you double check that it's installed? Otherwise I think conda install gdal should do the trick.

The HDF5 problem you had was probably related, conda install hdf5 should work in this case I believe.

[stichbury]

Thanks @astrojuanlu -- I don't use micromamba so wasn't sure how to work with:

micromamba create -n kedro38-dev python=3.8 hdf5 gdal "numpy=1.22.4" -c conda-forge -y

I have an existing conda env for Python 3.8 so have included hdf5 with export HDF5_DIR=/opt/homebrew/opt/hdf5 and export BLOSC_DIR=/opt/homebrew/opt/c-blosc as per some guidance on StackOverflow, and now I've added gdal and successfully done pip install -e ".[docs]" to completion.

1. It may be obvious, but is there any reason these can't be added to the requirements list somewhere, or they need to be included separately?
2. The make build-docs command is still building with 72 warnings e.g.

/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.biosequence.BioSequenceDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.dask.ParquetDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.geopandas.GeoJSONDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.holoviews.HoloviewsWriter
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.matplotlib.MatplotlibWriter
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.pandas.GBQQueryDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.pandas.GBQTableDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.pillow.ImageDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.redis.PickleDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.spark.DeltaTableDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.spark.SparkDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.spark.SparkHiveDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.spark.SparkJDBCDataSet
/Users/jo_stichbury/Documents/GitHub/kedro/docs/build/kedro.datasets.rst.rst:10: WARNING: autosummary: failed to import kedro.datasets.tensorflow.TensorFlowModelDataset
/Users/j

[stichbury]

Sorry to keep asking. I can't really help myself here as I have no idea how to make this work and it's important to me to be able to build the docs locally, so I want to be sure I can do so.

[astrojuanlu]

I'm sorry you're having a rough time getting this to work locally 🙏🏽

Sorry to keep asking. I can't really help myself here as I have no idea how to make this work and it's important to me to be able to build the docs locally, so I want to be sure I can do so.

Absolutely - and my top goal is that you can do it effectively and autonomously, without having to rely on our CI/RTD or Gitpod to do so.

I don't use micromamba so wasn't sure how to work with:

Sorry, I should have explained that better - micromamba is a drop-in replacement for conda, so in your case it would be

conda create -n kedro38-dev python=3.8 hdf5 gdal "numpy=1.22.4" -c conda-forge -y

It may be obvious, but is there any reason these can't be added to the requirements list somewhere, or they need to be included separately?

It's not obvious at all, these are classical Python packaging shenanigans every user has to go through at least once in their lifetime 😄 let me explain:

pip install can bring 2 types of packages: precompiled "wheels", which usually bundle everything that's needed (how this actually works is out of scope for this explanation) and source distributions, that contain only source code that needs to be compiled. For many packages in the data science ecosystem, this compilation is complex and requires heaby non-Python dependencies, like GDAL, HDF5, and others.

When there's no precompiled wheel for the architecture (in your case, M1 Mac) and Python version (say 3.8), pip is forced to download a source distribution and try to compile it, and the user needs to figure out how to get the non-Python dependencies installed without pip. For this there's a variety of techniques, and 3 pages of Google results might suggest 40 different ways of doing it, but essentially on Mac you can use Homebrew or conda.

I have an existing conda env for Python 3.8 so have included hdf5 with export HDF5_DIR=/opt/homebrew/opt/hdf5 and export BLOSC_DIR=/opt/homebrew/opt/c-blosc as per some guidance on StackOverflow, and now I've added gdal and successfully done pip install -e ".[docs]" to completion.

Good! ✔️ Although I'd think that we should avoid mixing Homebrew and conda dependencies.

The make build-docs command is still building with 72 warnings e.g.

Looks like there are still some missing dependencies

I think it will be better that we pair synchronously on this one to get to a working system more quickly, we might find things we could improve in our contribution docs.

[astrojuanlu]

Turns out we've accumulated a bunch of broken references in the past few days 😬 better to merge this PR ASAP.