[RFC] Move developer docs to main docs (or at least to an .rst tree)

[phlax]

description

At the moment there are a lot of useful dev docs/info spread through files in the repo.

I believe there are several advantages to centralizing the developer docs and bringing them into the main documentation

• we have various ci/checks etc that are only applied to the doc/rst tree
• centralizing docs would decrease the learning time/curve for new developers to find relevant documentation/adopt related standards etc
• one set of documents to maintain (altho i think there is an argument for maintaining the dev docs in a separate tree - but then 2)

we could still leave stubs/links in README files or similar

perhaps this pertains differently to code documentation in the actual source etc than to eg ci/tooling/sandbox docs - but even then there maybe a similar argument to collate the developer docs.

[phlax]

if noone has any objection to this idea, im happy to take it on

[mattklein123]

I agree this would be great to move away from MD into a single sphinx tree.

My main question is whether it should be an independent sphinx tree, or possibly a separate entry point / index. Meaning, I'm not sure we want end-users to see the dev docs. Even if we build as a single tree can we have different entry points / index for them?

[phlax]

I'm not sure we want end-users to see the dev docs.

yep, thats kinda my thought too

Even if we build as a single tree can we have different entry points / index for them?

rst has a master doc/single root model, so i think we would need to build as a separate doc tree.

If we publish the dev docs separately then there is also a question of where we publish them to.

[phlax]

one downside that im wondering about regarding moving .md -> .rst is that developers might be more inclined to quickly scribble a note in an md file in the code than write properly structured docs that are "published", but then having lots of scribbled notes is partly the problem, i think

[mattklein123]

rst has a master doc/single root model, so i think we would need to build as a separate doc tree.

We should research this fully, because I do think there may be value in ref linking from dev docs to main docs, though not a requirement.

If we publish the dev docs separately then there is also a question of where we publish them to.

I think we can publish them to the same site, just under a different path.

one downside that im wondering about regarding moving .md -> .rst is that developers might be more inclined to quickly scribble a note in an md file in the code than write properly structured docs that are "published", but then having lots of scribbled notes is partly the problem, i think

I agree, and RST is really not very much different from MD. Also, if we do make it a separate sphinx tree that can be built independently and doesn't require bazel, etc. it should build extremely quickly so I don't think the overhead is a huge deal.

[phlax]

im thinking there are 2 advantages to having a single tree

• internal linking (and related CI)
• versioning - we get versioned dev docs out of the box, and the relevant dev docs are together with the user docs

[mattklein123]

I agree on a single tree. I think the main issue is whether we can someone have a separate landing page.

For example, could we have the dev docs be hidden from the main TOC, link to them directly, and then the rest of it would show up fine?

[phlax]

a separate landing page.

ah, i see. i guess we would need to think how it should be in terms of navigation here https://www.envoyproxy.io/docs

but then yep, i think we can present the tree as we want in terms of entrypoints

the dev docs would still be visible to end users, and vice versa

[phlax]

For example, could we have the dev docs be hidden from the main TOC,

reading your question more carefully - no - i dont think we can hide parts of the tree. I may be wrong about that, but a quick search doesnt seem to suggest so

[mattklein123]

What about the hidden toc tree approach here? https://github.com/envoyproxy/envoy/blob/master/docs/root/intro/intro.rst#L14-L20.

(I think also that could be replaced with your redirect system, but am wondering if we could use that to have a well known path into the tree which doesn't show up in the main index)

[phlax]

i think i follow - 2 roots both with all pages, but a different set of hidden tocs - might work - i can do some testing to try and figure it out

there is a fundamental conceptual problem in my head - the links wont work out-of-the-box - ie internal links will be on the same url/path - but in reality not - i think it would duplicate the entire doc trees (maybe that is why you mention redirects)

[phlax]

reading the tocs docs more (https://www.sphinx-doc.org/en/1.5/markup/toctree.html) im thinking it would make those pages accessible - ie you could browse to the other pages, even tho they were not included in any tocs

not sure whether redirects would override an existing page, but i can check

(also checked exclude_paths but afaict that wont work)

[mattklein123]

^ Yup exactly. I think this would work out well. The only downside I can see is how long the docs take to test build, but that's a different issue you are tracking, and it should get a lot better once v2 is gone and we can cleanup a bunch of debt there. cc @htuch

[ggreenway]

I strongly agree that moving the docs to a single location is a good change. I've thought about doing this before, but never carved out time to do it.

[htuch]

I agree that we should move to the main docs. I don't think eliminating v2 will make things much faster, maybe 30% would be my guess, the real bottleneck (as I think I mentioned in our other thread on this) is Sphinx itself and the single-threaded docs build (GIL), we should try and figure out how to get it to work in its multiprocessing mode.

[moderation]

I did a short test on my 6 core 12 thread Ubuntu 20.04 laptop. I can confirm that without modification only a single thread processes the sphinx-build line in env SPHINX_SKIP_CONFIG_VALIDATION=true docs/build.sh.

Normal run:

time sphinx-build -W --keep-going -b html "${GENERATED_RST_DIR}" "${DOCS_OUTPUT_DIR}"

real    1m16.938s
user    1m14.809s
sys     0m0.736s

With -j auto turned on:

time sphinx-build -j auto -W --keep-going -b html "${GENERATED_RST_DIR}" "${DOCS_OUTPUT_DIR}"

Running Sphinx v3.2.1
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 838 source files that are out of date
updating environment: [new config] 838 added, 0 changed, 0 removed
reading sources... [100%] version_history/version_history
WARNING: the sphinxcontrib.httpdomain extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit
WARNING: doing serial read
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] version_history/v1.13.0 .. version_history/version_history
waiting for workers...
generating indices...  genindex http-routingtabledone
writing additional pages...  searchdone
copying images... [100%] _static/docker_compose_front_proxy.svg
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build finished with problems, 2 warnings.
Copying tabs assets
Writing redirects...
(good) intro/arch_overview/http/websocket.html --> intro/arch_overview/http/upgrades.html
(good) configuration/observability/access_log.html --> intro/arch_overview/observability/access_logging.html
(good) install/building.html --> start/building.html
(good) install/ref_configs.html --> start/install/ref_configs.html
(good) install/sandboxes/local_docker_build.html --> start/install/sandboxes/local_docker_build.html
(good) install/tools/config_load_check_tool.html --> start/install/tools/config_load_check_tool.html
(good) install/tools/route_table_check_tool.html --> start/install/tools/route_table_check_tool.html
(good) install/tools/schema_validator_check_tool.html --> start/install/tools/schema_validator_check_tool.html
(good) install/tools/tools.html --> start/install/tools/tools.html
(good) install/install.html --> start/start.html

real    0m46.635s
user    1m31.080s
sys     0m3.448s

The script emits a warning but multiple cores are engaged and the run does look a fair bit faster. Doesn't seem like there is a downside to using -j auto. Keep in mind that the Sphinx part of the docs build is the shortest part of the the overall docs build which is primarily taken up by the prior Bazel components. I've also noted the reading sources... invocation is still single threaded and quite slow, it is only the writing output... step that gets a boost.

The other thread @htuch refers too

[htuch]

Thanks @moderation. I think the Bazel phase might be a lot longer on less powerful machines (I should stop comparing to my 72-core build machine). Let's just turn on -j auto if it works without error; last time I tried it complained about multiprocessing unsafe Sphinx extensions and errored out.

The Bazel-side build time should drop by 50% when we get rid of v2.