LaTeX is not preserved in TOC text

[ComFreek]

Problem

TOC entries for headers containing inline LaTeX are wrongly generated:

1. [$\\mathrm{gcd}(a, b)$](#mathrmgcda-b)

# $\mathrm{gcd}(a, b)$

Here, it should have been 1. [$\mathrm{gcd}(a, b)$](#mathrmgcda-b).
I am using Markdown All in One configured with "markdown.extension.math.enabled": false.

I am using the latest dev build as of 2021-02-17.

[Lemmingh]

This change is by design.

It would take a too large amount of effort to keep the result safe and not aggressively escape backslash at the same time.

vscode-markdown/src/toc.ts

Lines 41 to 45 in c359515

	/**
	* The **rich text** (single line Markdown inline without raw HTML) representation of the rendering result (in strict CommonMark mode) of the heading.
	* This must be able to be safely put into a `[]` bracket pair without breaking Markdown syntax.
	*/
	visibleText: string;

vscode-markdown/src/toc.ts

Lines 447 to 467 in c359515

	/**
	* Extracts those that can be rendered to visible text from a string of CommonMark **inline** structures,
	* to create a single line string which can be safely used as **link text**.
	*
	* The result cannot be directly used as the content of a paragraph,
	* since this function does not escape all sequences that look like block structures.
	*
	* We roughly take GitLab's `[[_TOC_]]` as reference.
	*
	* @param raw - The Markdown string.
	* @param env - The markdown-it environment sandbox (**mutable**).
	* @returns A single line string, which only contains plain textual content,
	* backslash escape, code span, and emphasis.
	*/
	function createLinkText(raw: string, env: object): string {
	const inlineTokens: Token[] = commonMarkEngine.engine.parseInline(raw, env)[0].children!;
	return inlineTokens.reduce<string>((result, token) => {
	switch (token.type) {
	case "text":
	return result + token.content.replace(/[&*<>\[\\\]_`]/g, "\\$&"); // Escape.

[yzhang-gh]

Guess we need to add a patch for this. From my experience, many users will be affected by this.

[Lemmingh]

Rechecked.

Unsolvable

[Lemmingh]

I understand the need of displaying pretty math in headings and TOC, but I'm sorry,

The behavior cannot be changed anymore, unless we take the risk of putting broken links into TOC.

---

With #176, #194, #531, #540, #552, #570, #862, etc., we eventually lose control of how users embed math in Markdown, and have no reliable means of identifying math area.

We have to perform CommonMark only parsing when generating link text for TOC visible text.

[yzhang-gh]

I mean we need to be consistent.

• At the very least, if the math.enabled is true, we should make sure $...$ parts are still valid math envs (as this is the only syntax we support now).
• If the math.enabled is false, as you said, we don't know whether there are other kinds of math envs like \[\] etc. But I prefer to at least leave $...$ untouched.

By saying a patch, I mean, e.g., a regexp replacing/"protecting" the backslashes in $...$. This makes the solution not "neat", but otherwise we will see more issues complaining about this after v3.5.0 release.

BTW, do you have some examples where we must escape the \? I cannot think of some.

[Lemmingh]

The Even-odd Problem.

The backslash \ is the very escaping indicator.

---

vscode-markdown/src/toc.ts

Line 467 in c359515

return result + token.content.replace(/[&*<>\[\\\]_`]/g, "\\$&"); // Escape.

Look at them. This is the minimal escaping set.

• & : Entity or numeric character reference.
• * and _ : Emphasis.
• < and > : Autolink.
• [ and ] : Link or image.
• ` : Code span.
• \ : Escape.

Their occurrences can change semantics, and eventually break the link.

They have to be escaped. My comments in code are clear enough.

---

A simple example:

If you're going to display

<a href="#uri">\[</a>

Then, you'll have to write

[\\\[](#uri)

[Lemmingh]

A math area is a special variant of code span/block, from the perspective of a parser.

Among solutions, I think GitLab is the wisest, and I use it in my daily work.

GitLab actually does not introduce new syntax, instead, it reuses the existing syntax in CommonMark. A math area on GitLab is exactly a code span/block.

---

On GitLab,

[[_TOC_]]

# $``[(a+b)!]^2``$

````math
\frac{1}{2}
````

gives you

<ul class="section-nav"><li><a href="#ab2">[(a+b)!]^2</a></li></ul>
<h1 data-sourcepos="3:1-3:18" dir="auto">
<a id="user-content-ab2" class="anchor" href="#ab2" aria-hidden="true"></a><span data-math-style="inline"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mo stretchy="false">[</mo><mo stretchy="false">(</mo><mi>a</mi><mo>+</mo><mi>b</mi><mo stretchy="false">)</mo><mo stretchy="false">!</mo><msup><mo stretchy="false">]</mo><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">[(a+b)!]^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">[</span><span class="mopen">(</span><span class="mord mathdefault">a</span><span class="mspace" style="margin-right:0.2222222222222222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222222222222222em;"></span></span><span class="base"><span class="strut" style="height:1.064108em;vertical-align:-0.25em;"></span><span class="mord mathdefault">b</span><span class="mclose">)</span><span class="mclose">!</span><span class="mclose"><span class="mclose">]</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141079999999999em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span>
</h1>
<span data-math-style="display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow><annotation encoding="application/x-tex">\frac{1}{2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:2.00744em;vertical-align:-0.686em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.32144em;"><span style="top:-2.314em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">2</span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.686em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></span>

[Lemmingh]

I prefer to at least leave $...$ untouched.

It is not possible.

# $[(a+b)!](c+d)^2$

With pandoc-syntax, this is a math area.
Without, this is an inline link.

[Lemmingh]

vscode-markdown/src/toc.ts

Lines 41 to 45 in c359515

	/**
	* The **rich text** (single line Markdown inline without raw HTML) representation of the rendering result (in strict CommonMark mode) of the heading.
	* This must be able to be safely put into a `[]` bracket pair without breaking Markdown syntax.
	*/
	visibleText: string;

On my draft, it was

/**
 * The **single line plain text** representation of the rendering result (in CommonMark mode) of the heading.
 * This must be able to be safely put into a `[]` bracket pair as **link text** without breaking Markdown syntax.
 */
visibleText: string;

the same as GitLab's [[_TOC_]].

Later, I spent a hard time in createLinkText() to maintain backward compatibility. I think the current implementation is the best we can do, already the maximum backward compatibility.

[yzhang-gh]

I agree with you in theory (although I'm not very convinced by the \\\[ example which is a bit "unrealistic"...).

The problem is then we can only advocate the users to use the GitLab style math syntax? (I'm not against it but not sure whether other users will like it.)

---

Let's keep the change and leave this issue open to collect more feedback.

[coin8086]

So the time is April 23, 2023 now. Do we have any workaround for this problem? I'm using Markdown All in One v3.5.1 in VS Code. I got generated TOC text

[ABC $\\mathbf{A} \\mathbf{x}$](#abc-mathbfa-mathbfx)

for

### ABC $\mathbf{A} \mathbf{x}$

The escaped text in TOC is ugly. I'd rather have them removed from TOC, like

[ABC](#abc)

[yzhang-gh]

any workaround

Unfortunately no.

I don't have sufficient time to change this. Might need some help from the community.

[coin8086]

any workaround

Unfortunately no.

I don't have sufficient time to change this. Might need some help from the community.

Sorry to hear that. Thanks for your effort anyway!