Allow including nested captions in toctree function provided to HTML theme templates

[pradyunsg]

Is your feature request related to a problem? Please describe.
I'm exploring ways to enable more flexible/complex site structures with Sphinx, as part of a Sphinx theme (https://github.com/pradyunsg/lutra/). One of the useful bits of toctree metadata, that Sphinx does not include in the final markup, is all of captions for the not-top-level toctree items.

Currently, without this, certain navigational structures cannot be implemented -- eg: having top-level documents as tabs on the site and having their inner toctrees presented with captions in the sidebar (when that tab is active). Providing the nested toctree captions in the toctree directive would make it possible to implement such navifation structures.

Describe the solution you'd like

Allow specifying a captions_till_depth argument, defaulting to 1, on the toctree context variable that behaves like maxdepth but trims captions instead. The toctree generation code would then be adapted to both (a) generate the toctree captions in the nested bullet list and (b) trim them based on this variable.

Describe alternatives you've considered

1. Maintaining status quo -- this would prevent themes from providing support for certain kinds of site navigation designs.
2. Add a include_captions argument to the toctree context variable passed into the theme render context (and, add it to TocTree.get_toctree_for). This will default to False -- preserving the current default behaviour. However, when it is set to True, the final markup will contain the caption nested within the rest of the document.

Additional context

$ git clone https://github.com/pradyun/sphinx-nested-toctree-captions
$ cd sphinx-nested-toctree-captions
$ pip install sphinx-autobuild
$ sphinx-autobuild -b dirhtml . /tmp/build --open-browser

Look at the toctree presented. Note that it does not contain the inner-document's toctrees' captions (Aesthetics and Structural) but does contain the top-level-document's toctree's "Usage" caption. Screenshot of what is generated by alabaster:

[choldgraf]

We would definitely appreciate this in the PyData Sphinx Theme. We have exactly the use-case @pradyunsg describes. The top-level TOCtree items defined in the root_doc are "tabs" in a navigation header. Each second-level item has TOCtree items that show up in the sidebar only when the top-level page is active. We define captions at the second-level page, and have to do a really hacky re-engineer of the TOCtree generator here:

https://github.com/pydata/pydata-sphinx-theme/blob/d1103141d53cd5a86f6a00e6d8b141957db2a72e/src/pydata_sphinx_theme/__init__.py#L509-L523

This is both really inefficient since we have to do it twice, and also really hard to maintain because this is code most folks don't understand well.

Note that one extra challenge here is that this would effectively change the structure of the links underneath each caption. Currently if there's a caption for a toctree, it wraps the links in a <ul>, but for level 2+, it puts all toctree items as a flat list of <li> instead.

[pradyunsg]

Honestly, I’d even be fine if this wasn’t exposed in toctree, and just exposed in the underlying implementation.