feat(mdx-loader): the table-of-contents should display toc/headings of imported MDX partials

[anatolykopyl]

Note: @anatolykopyl started this PR, and then @slorber took it over.

Motivation

Fix #3915

It is now possible to import MDX partials and have headings of the partial appear in the TOC.

We'll consider this as a "new feature" instead of a bug fix, considering it remains an important behavior change, and an improvement over stock MDX. We'll release this in v3.2 instead of v3.1.1

---

Consider the following:

// _partial.mdx

## Partial heading

// doc.mdx

# Doc

## Doc heading

import Partial from "_partial.mdx"

<Partial/>

This doc now renders a TOC with 2 headings, while previously "Partial heading" didn't appear in the TOC.

In practice, the doc is now compiled as:

// doc.mdx
import Partial, 
  {toc as __tocPartial} from "_partial.mdx"

# Doc

## Doc heading

export const toc = [
  {
    "value": "Doc heading",
    "id": "doc-heading",
    "level": 2
  }, 
  ...__tocPartial,
]

Where __tocPartial contains {"value": "Partial heading","id": "partial-heading","level": 2}

This makes it possible to have headings working for "deeply-nested-partials" where one partial import another.

Notable change: if you write export const toc = ... in your MDX docs, we now never override this value anymore. Previously we did, but only when there were headings. We consider this as a bug fix and not a breaking change.

Test Plan

Unit tests + dogfood

Test links

https://deploy-preview-9684--docusaurus-2.netlify.app/tests/pages/partials-tests

Related issues/PRs

This PR adapts a similar fix from @dimaMachina made in Nextra to Docusaurus: shuding/nextra#2199

[netlify]

✅ [V2]

Name	Link
🔨 Latest commit	3d43631
🔍 Latest deploy log	https://app.netlify.com/sites/docusaurus-2/deploys/65aab464a3d8d3000833330f
😎 Deploy Preview	https://deploy-preview-9684--docusaurus-2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

⚡️ Lighthouse report for the deploy preview of this PR

URL	Performance	Accessibility	Best Practices	SEO	PWA	Report
/	🟠 83	🟢 98	🟢 100	🟢 100	🟠 89	Report
/docs/installation	🟠 89	🟢 98	🟢 100	🟢 100	🟠 89	Report
/docs/category/getting-started	🟠 76	🟢 100	🟢 100	🟢 90	🟠 89	Report
/blog	🟠 70	🟢 100	🟢 100	🟢 90	🟠 89	Report
/blog/preparing-your-site-for-docusaurus-v3	🟠 63	🟢 97	🟢 100	🟢 100	🟠 89	Report
/blog/tags/release	🟠 69	🟢 100	🟢 100	🟠 80	🟠 89	Report
/blog/tags	🟠 77	🟢 100	🟢 100	🟢 90	🟠 89	Report

[anatolykopyl]

I consider this PR done.
@slorber, could you please check it out? Any tips regarding how this can be improved are appreciated! I'd be glad to make changes to this PR if necessary.

[slorber]

That looks like good start thanks 👍

I'm not 100% fan of the implementation yet but overall the solution looks like what we want to adopt.

I'd like to have much more unit tests, testing more complex cases, such as using the same partial twice in a page, or importing the same partial using different names.

That would also be interesting to dogfood with more complex cases, such as a deeply nested tree of partials (having partials importing other partials)

[slorber]

toc0 is not imported here, is this expected?

[anatolykopyl]

Seems like we're going down the road of only modifying the AST and not the actual js, so this is expected.

[slorber]

I know the original code was already low-level, but I'd appreciate if we could make it more readable and high-level. I mean the intent of this algorithm should be clear without spending time trying to understand the implementation details. Currently it's a bit hard to see the big picture of it. I would extract things in smaller functions for example.

[anatolykopyl]

Moved the visitor for each type of node into its own function. This is a bit more readable now, but I can try to do some further reorganisation if needed.

[anatolykopyl]

Thanks for the detailed feedback! I'll be working on improving this PR in the following weeks.

[Josh-Cena]

I'm quite happy with this implementation; exactly how I pictured it.

[anatolykopyl]

Moved the dogfooding into _docs tests.
Added a deeply nested partial and a partial with props in the heading. Last one is not working, the heading shows the text passed to the prop as expected, but the ToC displays just the text props.propname. Quite interesting, no idea how this can possibly be fixed.

Also since I changed the plugin to not generate any js anymore, only ASTs, the existing unit tests for the ToC plugin became useless, as they checked generated mdx. I changed the unit tests to output ASTs, so the work of the plugin is visible.

[wooorm]

Last one is not working, the heading shows the text passed to the prop as expected, but the ToC displays just the text props.propname.

What output do you want, what would be “working”?

The code you use uses toValue: https://github.com/anatolykopyl/docusaurus/blob/1fdd6af8c79e7765ff4d7b4731e605d0d95c1316/packages/docusaurus-mdx-loader/src/remark/utils/index.ts#L76C17-L76C24.
Which calls toString() on an expression ({props.propname}): https://github.com/anatolykopyl/docusaurus/blob/1fdd6af8c79e7765ff4d7b4731e605d0d95c1316/packages/docusaurus-mdx-loader/src/remark/utils/index.ts#L99
Which returns node.value, which is props.propname.

[slorber]

Thanks, that looks like it's working nicely.

The {props.propname} is not a problem to consider for now so you can remove that test.

I understand why you changed the test snapshots to MDAST although I don't like it. I'll try to come up with a solution and commit a thing to your PR soon.

[slorber]

Thanks @anatolykopyl , looks like it's working nicely.

Almost ready, I'm just going to refactor some details and merge asap!