✨ NEW: Add Docutils MyST config and CLI (#426)

This commit updates the docutils only `Parser`, to include relevant MyST configuration options,
which can be used with the docutils API, CLI or via a `docutils.conf` file.

CLI commands are also exposed for docutils writers:
`myst-docutils-html`, `myst-docutils-html5`, `myst-docutils-latex`, `myst-docutils-pseudoxml`, `myst-docutils-xml`.

In addition,
the commit also stops mathjax classes being added to sections, when parsing in docutils mode,
and adds help metadata to the `MdParserConfig`attributes (for programmatic documentation).

Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>

docs/api/parsers.md

@@ -9,28 +9,34 @@
 
 The MyST Parser comes bundled with some helper functions to quickly parse MyST Markdown and render its output.
 
+:::{important}
+These APIs are primarily intended for testing and development purposes.
+For proper parsing see {ref}`myst-sphinx` and {ref}`myst-docutils`.
+:::
+
 ### Parse MyST Markdown to HTML
 
-For example, the following code parses markdown and renders as HTML:
+The following code parses markdown and renders as HTML using only the markdown-it parser
+(i.e. no sphinx or docutils specific processing is done):
 
 ```python
 from myst_parser.main import to_html
-to_html("some *text*")
+to_html("some *text* {literal}`a`")
 ```
 
 <!-- #region -->
 ```html
-'<p>some <em>text</em></p>\n'
+'<p>some <em>text</em> <code class="myst role">{literal}[a]</code></p>\n'
 ```
 <!-- #endregion -->
 
 ### Parse MyST Markdown to docutils
 
-The following function renders your text as **docutils objects** (for example, for use with the Sphinx ecosystem):
+The following function renders your text as **docutils AST objects** (for example, for use with the Sphinx ecosystem):
 
 ```python
 from myst_parser.main import to_docutils
-print(to_docutils("some *text*").pformat())
+print(to_docutils("some *text* {literal}`a`").pformat())
 ```
 
 ```xml
@@ -39,8 +45,17 @@ print(to_docutils("some *text*").pformat())
         some
         <emphasis>
             text
+
+        <literal>
+            a
 ```
 
+:::{note}
+This function only performs the initial parse of the AST,
+without applying any transforms or post-processing.
+See for example the [Sphinx core events](https://www.sphinx-doc.org/en/master/extdev/appapi.html?highlight=config-inited#sphinx-core-events).
+:::
+
 ### Parse MyST Markdown as `markdown-it` tokens
 
 The MyST Parser uses `markdown-it-py` tokens as an intermediate representation of your text.

docs/api/reference.rst

@@ -81,6 +81,16 @@ Additional Methods
 .. autofunction:: myst_parser.sphinx_renderer.mock_sphinx_env
 
 
+.. _api/docutils_parser:
+
+Docutils Parser Reference
+-------------------------
+
+.. autoclass:: myst_parser.docutils_.Parser
+    :members: parse
+    :undoc-members:
+    :member-order: bysource
+    :show-inheritance:
 
 .. _api/sphinx_parser:
 

docs/conf.py

@@ -4,6 +4,9 @@
 # list see the documentation:
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 
+from sphinx.application import Sphinx
+from sphinx.util.docutils import SphinxDirective
+
 from myst_parser import __version__
 
 # -- Project information -----------------------------------------------------
@@ -167,7 +170,34 @@ def run_apidoc(app):
 ]
 
 
-def setup(app):
+def setup(app: Sphinx):
     """Add functions to the Sphinx setup."""
+
+    class DocutilsCliHelpDirective(SphinxDirective):
+        """Directive to print the docutils CLI help."""
+
+        has_content = False
+        required_arguments = 0
+        optional_arguments = 0
+        final_argument_whitespace = False
+        option_spec = {}
+
+        def run(self):
+            """Run the directive."""
+            import io
+
+            from docutils import nodes
+            from docutils.frontend import OptionParser
+
+            from myst_parser.docutils_ import Parser as DocutilsParser
+
+            stream = io.StringIO()
+            OptionParser(
+                components=(DocutilsParser,),
+                usage="myst-docutils-<writer> [options] [<source> [<destination>]]",
+            ).print_help(stream)
+            return [nodes.literal_block("", stream.getvalue())]
+
     # app.connect("builder-inited", run_apidoc)
     app.add_css_file("custom.css")
+    app.add_directive("docutils-cli-help", DocutilsCliHelpDirective)

docs/docutils.md

@@ -0,0 +1,79 @@
+(myst-docutils)=
+
+# MyST with Docutils
+
+Sphinx, and thus MyST-Parser, is built on top of the [Docutils](https://docutils.sourceforge.io/docs/) package.
+MyST-Parser offers a renderer, parser and CLI-interface for working directly with Docutils, independent of Sphinx, as described below.
+
+:::{note}
+Since these tools are independent of Sphinx, this means they cannot parse any Sphinx or Sphinx extensions specific roles or directives.
+:::
+
+On installing MyST-Parser, the following CLI-commands are made available:
+
+- `myst-docutils-html`: converts MyST to HTML
+- `myst-docutils-html5`: converts MyST to HTML5
+- `myst-docutils-latex`: converts MyST to LaTeX
+- `myst-docutils-xml`: converts MyST to docutils-native XML
+- `myst-docutils-pseudoxml`: converts MyST to pseudo-XML (to visualise the AST structure)
+
+Each command can be piped stdin or take a file path as an argument:
+
+```console
+$ myst-docutils-html --help
+$ echo "Hello World" | myst-docutils-html
+$ myst-docutils-html hello-world.md
+```
+
+The commands are based on the [Docutils Front-End Tools](https://docutils.sourceforge.io/docs/user/tools.html), and so follow the same argument and options structure, included many of the MyST specific options detailed in [](sphinx/config-options).
+
+:::{dropdown}  Shared Docutils CLI Options
+```{docutils-cli-help}
+```
+:::
+
+The CLI commands can also utilise the [`docutils.conf` configuration file](https://docutils.sourceforge.io/docs/user/config.html) to configure the behaviour of the CLI commands. For example:
+
+```
+# These entries affect all processing:
+[general]
+myst-enable-extensions: deflist,linkify
+myst-footnote-transition: no
+
+# These entries affect specific HTML output:
+[html writers]
+embed-stylesheet: no
+
+[html5 writer]
+stylesheet-dirs: path/to/html5_polyglot/
+stylesheet-path: minimal.css, responsive.css
+```
+
+You can also use the {py:class}`myst_parser.docutils_.Parser` class programmatically with the [Docutils publisher API](https://docutils.sourceforge.io/docs/api/publisher.html):
+
+```python
+from docutils.core import publish_string
+from myst_parser.docutils_ import Parser
+
+source = "hallo world\n: Definition"
+output = publish_string(
+    source=source,
+    writer_name="html5",
+    settings_overrides={
+        "myst_enable_extensions": ["deflist"],
+        "embed_stylesheet": False,
+    },
+    parser=Parser(),
+)
+```
+
+Finally, you can include MyST Markdown files within a RestructuredText file, using the [`include` directive](https://docutils.sourceforge.io/docs/ref/rst/directives.html#include):
+
+```rst
+.. include:: include.md
+   :parser: myst_parser.docutils_
+```
+
+```{important}
+The `parser` option requires `docutils>=0.17`
+```

docs/index.md

@@ -100,6 +100,7 @@ syntax/reference
 :caption: Topic Guides
 explain/index.md
 sphinx/index.md
+docutils.md
 api/index.md
 develop/index.md
 ```

docs/sphinx/index.md

@@ -1,3 +1,5 @@
+(myst-sphinx)=
+
 # MyST with Sphinx
 
 The MyST Parser comes bundled with a Sphinx extension that allows you to use write Sphinx documentation entirely in MyST (or, in a combination of rST and MyST).

docs/sphinx/use.md

@@ -32,6 +32,20 @@ To include rST, we must first "wrap" the directive in the [eval-rst directive](s
 .. include:: snippets/include-rst.rst
 ```
 
+(howto/include-md)=
+## Include Markdown files into an rST file
+
+To include a MyST file within a ReStructuredText file, we can use the `parser` option of the `include` directive:
+
+```rst
+.. include:: include.md
+   :parser: myst_parser.sphinx_
+```
+
+```{important}
+The `parser` option requires `docutils>=0.17`
+```
+
 ## Use MyST in Jupyter Notebooks
 
 The [MyST-NB](https://myst-nb.readthedocs.io) tool provides a Sphinx extension for parsing **Jupyter Notebooks written with MyST Markdown**. It includes features like automatically executing notebooks during documentation builds, storing notebook cell outputs in order to insert them elsewhere in your documentation, and more. See the [MyST-NB documentation](https://myst-nb.readthedocs.io) for more information.