Add mathematics support to rustdoc #17390

huonw · 2014-09-19T12:56:48Z

Revival of #16991, fixes #16300.

This uses the new support in hoedown for parsing mathematics. Current supported delimiters are \$ ... \$ (text), \\[ ... \\] (display), $$...$$ (inferred). Hoedown currently requires that the first two have spaces between the delimiters and the mathematics, and that all of them have space after the delimiters (hoedown/hoedown#120).

The KaTeX library is packaged without modifications with the distribution, so this should work without an internet connection. It consists of ~150 KB of JS/CSS and nearly 850 KB of fonts, the latter can be cut to 200KB if we only include the WOFF files (i.e. remove the EOT and TTF ones). This would require editing the files slightly. KaTeX does not have nearly the same support as MathJax, but it is significantly faster and smaller (it would not be nearly as reasonable to package MathJax).

This also uses this support in a few selected places in the standard libraries.

General comments:

inlined documentation with mathematics requires a enable_math on the crates in which it is inlined for the maths to be rendered
I added basic support for standalone markdown files, but I'm not sure if we want that in this iteration. Standalone markdown files currently include nothing else by default, and this is inserting JS and CSS. Furthermore, this requires but doesn't enforce that a katex installation exists as a sibling of the rendered file (which is true if the markdown file is rendered into a directory with other crates, e.g. like the main docs).
I'm sick at the moment, so the code could be nonsense.
I am preserving @gankro's work; if they're ok with it, I will compress the history.

rust-highfive · 2014-09-19T12:56:53Z

Warning

These commits modify unsafe code. Please review it carefully!

huonw · 2014-09-19T13:02:48Z

Rendering:

http://huonw.github.io/rust/build/doc/rand/distributions/gamma/struct.Gamma.html (current)
http://huonw.github.io/rust/build/doc/rand/distributions/exponential/struct.Exp.html (current)
There's a variety of examples in collections, e.g. priority queue
Reexport http://huonw.github.io/rust/build/doc/std/collections/priority_queue/

Standalone example (source)

Gankra · 2014-09-19T13:16:13Z

Feel free to squash the commits.

alexcrichton · 2014-09-19T13:59:03Z

src/librustdoc/html/markdown.rs

+                    hoedown_buffer_put(ob, c as *const libc::c_void, close.len() as libc::size_t);
+                }
+            })
+        });


How different is this from the default hoedown math callback?

It just wraps it in \[ \] or  (for display vs. non-display).

alexcrichton · 2014-09-19T14:00:47Z

cc @brson about the licensing of LaTeX and friends.

Gankra · 2014-09-19T14:52:08Z

src/librustdoc/html/layout.rs

@@ -34,7 +36,11 @@ pub fn render<T: fmt::Show, S: fmt::Show>(
    dst: &mut io::Writer, layout: &Layout, page: &Page, sidebar: &S, t: &T)
    -> io::IoResult<()>
 {
-    write!(dst,
+    // Reset state on whether we've seen math, so as to avoid loading mathjax


Old reference to mathjax

Gankra · 2014-09-19T14:58:49Z

Does rustdoc have any particular "supported" browser range? If the fonts aren't there, how does it render?

Only having WOFF wouldn't be too bad

Gankra · 2014-09-19T15:10:40Z

hoedown 120 has been patched: hoedown/hoedown#125

steveklabnik · 2014-09-19T16:40:28Z

Excited to see this land someday...

sophiebits · 2014-09-19T17:58:55Z

In TeX and LaTeX, the $$ delimiter is used for display-style math only. That is,

This operation should compute in $$O(1)$$ time.

produces:

Perhaps it makes sense to use a different delimiter here for inline math?

Gankra · 2014-09-19T18:11:01Z

@spicyj in conformance with the general uh... style of markdown, $$ is a do the right thing operator. It will be inline when inline and display when a block. This is fine in my opinion, because in writing text, if you have display math inline, you are simply using bad style. I say this as an academic who uses latex for a living. You can use  and \[\] if you find this troubling.

SimonSapin · 2014-09-19T18:29:46Z

Does rustdoc have any particular "supported" browser range?

The HTML output contains:

<body class="rustdoc">
    <!--[if lte IE 8]>
    <div class="warning">
        This old browser is unsupported and will most likely display funky
        things.
    </div>
    <![endif]-->

xymostech · 2014-09-19T21:12:12Z

If you don't support IE 8 and below, you should definitely get rid of the EOTs. From caniuse, it looks like the main thing that supports ttf but not woff is old android browser versions.

huonw · 2014-09-19T23:30:58Z

In TeX and LaTeX, the $$ delimiter is used for display-style math only. That is,

Yep, I know, but all this parsing is provided by hoedown.

@gankro, thanks for keeping track (@jmendeth, thanks for the improvements!).

It seems WOFF is only supported by the very latest android versions, so the majority of android users can't see WOFF. So I'll definitely remove the EOTs but not the ttfs for now, @xymostech, is there a good way to make a 'custom' KaTeX build rather than just manually removing the relevant URLs from @font-face declarations in the CSS?

xymostech · 2014-09-20T01:09:33Z

There isn't at the moment. However, your question has reminded me how annoying it is to modify fonts.css, so I'm going to make that better sometime soon.

huonw · 2014-09-20T01:44:43Z

@xymostech thanks for the info! (Deleting the URLs is easy enough, so it's not a huge problem.)

mildsunrise · 2014-09-20T07:48:08Z

In TeX and LaTeX, the $$ delimiter is used for display-style math only. That is,

We know, but $...$ is too frequently used in texts so we chose to stick with Kramdown syntax which makes $$...$$ context sensitive.

There's another flag which enables $...$ and makes $$...$$ always produce display-style math, like in TeX.

tbu- · 2014-09-20T13:01:00Z

Could this maybe be done as a step of doc building instead of client-side in the browser? It seems KaTeX outputs HTML code which then visually resembles in the equations, so I don't see why it wouldn't be sensible to do it this way.

huonw · 2014-09-20T13:04:17Z

We would need a Rust LaTeX renderer (or bindings to one in another language). Shelling out to an external program does not work unless we package the entire program and its dependencies (it is also fairly slow, but equations are usually rare so performance is not the major issue).

Theoretically the renderer is just an implementation detail, so this can be changed in future.

mildsunrise · 2014-09-20T13:04:26Z

@tbu- KaTeX now exports a katex CLI, which would allow rustdoc to render every equation once, at build, in the math callback.

However, doing this would require users to install Node.JS before they can build the docs.

SimonSapin · 2014-09-20T13:04:29Z

@tbu-, I think rustdoc should do that when node.js happens to be available, but fall back to client-side rendering when it’s not to avoid making node.js a hard dependency.

huonw · 2014-09-20T13:05:20Z

(Optionally shelling out is another choice, of course.)

tbu- · 2014-09-20T13:07:09Z

@SimonSapin That sounds like a reasonable way.

mildsunrise · 2014-09-20T13:07:45Z

Packaging Node.JS is not an option I suppose, but you could add KaTeX as a submodule and do node ~/path/to/KaTeX/cli and render through that. If a node command is not available, fallback to rendering in the browser.

xymostech · 2014-09-22T21:28:34Z

@huonw I just added support for modifying the css files in a saner way. If you modify the variables at the top of static/fonts.less, you can opt-out of EOT support, and also change where you want the fonts to be located.

huonw · 2014-09-23T09:15:50Z

@xymostech awesome, thanks!

huonw · 2014-09-24T08:08:55Z

Updated. (I'm not keen on implementing the shelling-out-to-node thing right now, with our current IO libs, since it is being called from a C callback. It is just an implementation detail so I don't think this is particularly important.)

Unlike MathJaX, KaTeX is small enough to be included directly in the distribution; the JS is 110KB, the CSS is 20KB, and the full suite of fonts are about 850KB.

These are only used by IE8 and below, which we do not support.

Requires the KaTeX files to be arranged correctly.

Only on all pages in crates with #[doc(enable_math)], not on all pages in all crates.

huonw · 2014-09-29T12:15:57Z

Updated again, with some squashing.

rust-lang/hoedown#2 needs to be merged before this can land.

Also, I just noticed that hoedown does have this EXPLICIT_MATH option, that adds parsing for $...$ as inline math and switches $$...$$ to block only (i.e. not inferred). I don't know if we want to take this approach.

Gankra · 2014-09-29T12:50:50Z

@huonw I avoided the EXPLICIT_MATH option because it would probably play poorly with macro docs or financial libraries.

Edit: although in the former case I suppose the $'s should always be fenced?

mildsunrise · 2014-09-29T15:00:56Z

@huonw @gankro I'd also recommend sticking with Kramdown (inferred) syntax, it's more accepted.

Gankra · 2014-10-04T14:08:40Z

So what's the status of this after the resounding "meh" this got from the core team meeting? Is that a polite death mark, or does it just need a stronger motivating write up? Or is it fine?

alexcrichton · 2014-10-07T14:54:08Z

@gankro, @huonw, beyond scratching an itch, does supporting pretty math documentation support a more general purpose? I'm growing more wary of the standard rustdoc distribution not being able to interoperate with other markdown-like systems.

I'd love to be able to support some sort of functionality like this as an extension to rustdoc! We just don't quite have the infrastructure for a modification such as that at this time...

Gankra · 2014-10-07T16:45:11Z

It allows us to write better docs (for readers) in an easier way (for writers). Not much else to it. What did you have in mind for "other markdown-like systems"?

mildsunrise · 2014-10-07T16:50:06Z

Again, math syntax is not Hoedown-specific. It comes from the Kramdown spec (and parser) and we could say it's "standard" in the Markdown world.

alexcrichton · 2014-10-07T18:53:58Z

@gankro I don't disagree that this does make the docs a little better, but it seems like a very small delta for the size of this change (bundling the katex project for example). The readability improvement, while larger than 0, is much smaller than, for example, hyperlinking types/methods in documentation (although that would certainly be a larger undertaking).

@jmendeth true! I suppose I don't personally really see a strong argument either not in favor or in favor for this, and erring on the side of adding whatever we can is probably not the right way to go.

huonw · 2014-10-31T00:35:41Z

We're not really sure the direction rustdoc wants to go, so the team is understandably nervous about enabling more markdown extensions and hence I'm not going to push for this to be merged now. Thanks to @jmendeth and @xymostech for your help, hopefully we can revisit in future and use your wonderful work.

In any case, I think there is a slightly reasonable way to do this without direct rustdoc support via inline HTML which I will investigate and write up if it works.

Gankra · 2014-10-31T01:39:29Z

RIP 😢

mildsunrise · 2014-10-31T10:56:43Z

:-/

nixpulvis · 2016-08-27T15:10:49Z

Can we talk about making this an experimental feature, I think there is a real use case for this.

mgeisler · 2017-10-25T20:08:51Z

Again, math syntax is not Hoedown-specific. It comes from the Kramdown spec (and parser) and we could say it's "standard" in the Markdown world.

I've never heard of kramdown before, but I know LaTeX :-) I believe that is the standard that's actually interesting to mimic, also when it comes to the delimiters. Basically, $x$, and $y$ is kind of readable after some years of practice with LaTeX, but $$x$$, and $$y$$ is much more heavy to my eyes.

So if this ever gets revived (which would be great), then I would vote for making it possible on to select the EXPLICIT_MATH option mentioned above and have the traditional delimiters.

jannschu · 2022-05-20T12:10:33Z

GitHub now has math support, also part of GFM spec. Maybe this spec be used here as well?

alexcrichton reviewed Sep 19, 2014
View reviewed changes

Gankra reviewed Sep 19, 2014
View reviewed changes

Gankra added 2 commits September 29, 2014 21:59

update to latest hoedown to get builtin mathjax support

035a321

rustdoc: enable support for mathematics.

4a1b2e8

huonw added 8 commits September 29, 2014 21:59

Avoid being specific to mathjax.

0bbb5ac

Add KaTeX to the distribution.

a3b23a1

Switch to KaTeX for math.

84e1804

Unlike MathJaX, KaTeX is small enough to be included directly in the distribution; the JS is 110KB, the CSS is 20KB, and the full suite of fonts are about 850KB.

Remove the EOT fonts from KaTeX.

71052f8

These are only used by IE8 and below, which we do not support.

Use LaTeX in a few of the standard lib crates.

ca92372

Allow LaTeX in standalone markdown files via --markdown-enable-math.

2088184

Requires the KaTeX files to be arranged correctly.

Semi-conditionally insert the CSS.

8a0aab1

Only on all pages in crates with #[doc(enable_math)], not on all pages in all crates.

Update hoedown.

98e6f14

huonw force-pushed the mathjax branch from 7c8cf06 to 98e6f14 Compare September 29, 2014 12:12

huonw closed this Oct 31, 2014

steveklabnik mentioned this pull request Dec 11, 2014

Add Mathjax (KaTeX?) Support to Rustdoc #16300

Closed

alilleybrinker mentioned this pull request Jun 13, 2016

Add LaTeX support to Rustdoc #34261

Closed

vks mentioned this pull request Nov 8, 2017

Documentation should render math properly vks/special-fun#6

Open

497e0bdf29873 mentioned this pull request Oct 24, 2022

Rustdoc does not correctly ignore input inside HTML tags. #103473

Closed

marsella mentioned this pull request Jan 24, 2023

Add nonce masking operation to PaillierEncryptionKey boltlabs-inc/tss-ecdsa#124

Merged

Add mathematics support to rustdoc #17390

Add mathematics support to rustdoc #17390

Conversation

huonw commented Sep 19, 2014

rust-highfive commented Sep 19, 2014

huonw commented Sep 19, 2014

Gankra commented Sep 19, 2014

alexcrichton Sep 19, 2014

Choose a reason for hiding this comment

huonw Sep 19, 2014

Choose a reason for hiding this comment

alexcrichton commented Sep 19, 2014

Gankra Sep 19, 2014

Choose a reason for hiding this comment

Gankra commented Sep 19, 2014

Gankra commented Sep 19, 2014

steveklabnik commented Sep 19, 2014

sophiebits commented Sep 19, 2014

Gankra commented Sep 19, 2014

SimonSapin commented Sep 19, 2014

xymostech commented Sep 19, 2014

huonw commented Sep 19, 2014

xymostech commented Sep 20, 2014

huonw commented Sep 20, 2014

mildsunrise commented Sep 20, 2014

tbu- commented Sep 20, 2014

huonw commented Sep 20, 2014

mildsunrise commented Sep 20, 2014

SimonSapin commented Sep 20, 2014

huonw commented Sep 20, 2014

tbu- commented Sep 20, 2014

mildsunrise commented Sep 20, 2014

xymostech commented Sep 22, 2014

huonw commented Sep 23, 2014

huonw commented Sep 24, 2014

huonw commented Sep 29, 2014

Gankra commented Sep 29, 2014

mildsunrise commented Sep 29, 2014

Gankra commented Oct 4, 2014

alexcrichton commented Oct 7, 2014

Gankra commented Oct 7, 2014

mildsunrise commented Oct 7, 2014

alexcrichton commented Oct 7, 2014

huonw commented Oct 31, 2014

Gankra commented Oct 31, 2014

mildsunrise commented Oct 31, 2014

nixpulvis commented Aug 27, 2016

mgeisler commented Oct 25, 2017

jannschu commented May 20, 2022