doc,tools: Replace client-side syntax highlighting for API docs with compile-time syntax highlighting

[tolmasky]

This PR removes the need to do syntax highlighting at runtime whenever a user loads the API docs by instead doing it during the generation of the documentation. I pulled these changes out of the larger RunKit docs changes since these have been ready to go for a while and I think we could benefit from them now (you can read more here). Originally this change made the RunKit changes significantly simpler to implement, but it has a number of benefits on its own:

1. The page will load much faster since there is much less JavaScript taking place.
2. Users with slow connections or JavaScript turned off will still see highlighted code.

To expand a bit more and bring some of the information from the other PR into here:

1. This change keeps all the fundamental mechanics of the current highlighting system:
   a. We still use highlightjs to produce the markup.
   b. We use the same theme.
   c. Syntax highlighting is triggered in the same way from markdown.
2. Instead of committing a vendored copy of the bundled highlightjs code that is loaded on the client and run after load, we instead include highlightjs in the build tools and run it at documentation generation time. For pages like /docs/api/all.html, where the client highlightjs has to highlight 27,325 code examples, I think this makes a dramatic difference in the experience (not to mention the electricity we'll be saving spread across all the computers that load these pages).
3. One additional nice side-effect is that the tool will now warn when you create a code block without a language specified, leading to a reduction of accidental unhighlighted cases in the future. I know a lot of work recently went in to manually spot these in the existing docs, so it'll be nice to keep things at 100% highlighted going forward.
4. I have made the allhtml.js code a little less brittle by specifically looking for a marker in the code instead of relying on the regex for establishing the correct boundary.

I'm happy to get this back-ported to earlier versions too if someone can direct me to the right way to do that.

• documentation is changed or added (technically only generated documentation is changed)
• commit message follows commit guidelines

[Trott]

I'm surprised npm didn't insert this in alphabetical order. Are you using an old version of npm possibly?

[tolmasky]

I think this is my fault from doing a rebase/merge. I'll put it in the right place and re-push.

[Trott]

Ah, a rebase, that makes sense. It doesn't much matter. I was just wondering how it happened.

[tolmasky]

I went ahead and fixed it (and rebased again) -- that way if someone else adds a package they won't be confused having it move around.

[DerekNonGeneric]

Probably an oversight, but the highlightjs package is now deprecated, so you'd want the newer highlight.js one instead.

[tolmasky]

👍 Good catch! I went ahead and changed it to highlight.js. Thanks!

[Trott]

Seems like a good idea to me, but wondering if @DerekNonGeneric has thoughts they'd like to share.

[DerekNonGeneric]

I've not looked at the other PR referenced since it doesn't seem related, but please clarify if I'm mistaken.

---

What you are doing in this PR is something that I had considered and decided against. It's also not the way syntax highlighting was being performed in the past.

The page will load much faster since there is much less JavaScript taking place.

I don't believe this to be true, but there is a way to confirm. Have you compared the size of the highlight.js bundle to the size difference of the HTML documents before and after you perform SSR of the syntax highlighting? If the differences are larger than the bundle, they will generally take longer to download (there is also HTTP request times to take into account).

Users with slow connections or JavaScript turned off will still see highlighted code.

The point I've made above affects users w/ slow connections, and almost nobody has JavaScript turned off (especially in the target audience of these documents).

---

Doing what is proposed in this PR will bloat the HTML files, which is why syntax highlighting is a good candidate for client-side JavaScript. Unless I'm missing something, I don't believe these changes are beneficial, however, it's possible that don't understand the motivation behind them.

I'd also like to add that I don't really hold too strong of an opinion on the matter, so no big deal if others have a preference.

[tolmasky]

It's also not the way syntax highlighting was being performed in the past.

Perhaps there is some confusion here (very much possibly on my end), but the only point I was making here is that this change doesn't change the syntax highlighter -- it only changes when the highlighting is done. Both use highlightjs and thus should generate the same final HTML and require no changes to CSS, etc. In other words, no one should have to change anything in response to this change.

If the differences are larger than the bundle, they will generally take longer to download (there is also HTTP request times to take into account).

From my own tests on a MacBook 2.3 GHz, all.html loads half a second faster with this change, and almost the entirety of this can be traced to the fact that it takes a full 450ms+ for the syntax highlighting to currently sequentially pop in each individual example. This is also particularly noticeable on mobile.

This is at the cost of a difference of 32KB in gzipped size (729KB for inlined, 697KB for original). Which, depending on cache, is further offset by highlight.pack.js which is 12KB zipped and not present in the new version. So a worst case scenario difference of 32KB increase over the wire, and best case scenario difference of 20KB over the wire. This is why the profile difference is dominated by the JavaScript execution vs. byte transmission.

The individual documentation files are even less of a concern. For example, async_hooks.html has the exact same gzipped size of 16KB, and crypto.html is only 1KB larger.

One other benefit to this is that people can now freely include whatever language they want in the markdown examples without having to recompile highlight.pack.js if they include a language that wasn't previously used.

[tolmasky]

Just wanting to close the loop on this -- there are no other changes you want me to make right?

[addaleax]

Fwiw, I also don’t have a strong preference here, but I’m okay with landing this and keeping the possibility of changing it back open if somebody complains or longer page load times do turn out to be problematic.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/32236/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/32243/

[tolmasky]

Just following up here again, not sure if what is needed is approval from all the reviewers listed on the right or if anything else is required from me.

[DerekNonGeneric]

@tolmasky, I've removed my thumbs down in case that might help you get this landed. :)

[addaleax]

Yeah, thanks for the ping. I’ve kicked off a new CI run. 👍

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/32320/

[tolmasky]

Is passing CI a requirement for this commit to go in? I was under the impression that the main branch was failing/flaky generally (just my impression from here: https://github.com/nodejs/node/commits/master). Should I rebase to get this on top of the latest stuff to see if that helps? Or is CI automatically attempting the build as if it was merged on top of the main branch?

[DerekNonGeneric]

@tolmasky, would you mind explaining what this meta tag will be used for (if anything)?

[DerekNonGeneric]

The problem it solves is that there is currently no easy way to determine what version of the docs you are viewing without falling back regex hacks on the title or h1 element, which should probably not be relied on as a future-proof method, especially when localizing the docs. The URL itself also isn't reliable since these docs often live at vXX-latest. This meta tag provides a place you can reliably query the version of the node docs, whether as a script on the page (as in the other commit), or a program operating on the page (for example if you are running end-to-end tests with puppeteer), or even when visually comparing a diff.
— #34148 (comment)

I wonder if this meta tag would be useful for @zeke's localization work.

[addaleax]

Is passing CI a requirement for this commit to go in?

@tolmasky Green or yellow – so yes, this is ready and I’ll go ahead and merge it. (Yellow means known flaky tests failed, which is okay.)

I know @DerekNonGeneric just posted another comment, but it doesn’t seem like something that needs to be figured out before merging. (As mentioned above, merging something doesn’t need to mean that the conversation has come to an end.)

I was under the impression that the main branch was failing/flaky generally (just my impression from here: https://github.com/nodejs/node/commits/master).

That impression is correct, unfortunately 😕

Should I rebase to get this on top of the latest stuff to see if that helps? Or is CI automatically attempting the build as if it was merged on top of the main branch?

Yes, CI automatically rebases, so nothing to do there – rebasing can be helpful if it resolves merge conflicts or it’s a very outdated PR, but otherwise, it can obscure whether new changes have been added that would need to be reviewed and tested separately or not – so no rebasing is definitely okay here.

[addaleax]

Landed in fa2e460

[tolmasky]

Hi @DerekNonGeneric, I wanted to make sure I answered your question:

This is a hold-over from the previous commit, but I kept it in because it is generally useful for working in this part of the code base, both for debugging and also for consumers of the page (for example in the companion scripts I wrote to test my changes).

The problem it solves is that there is currently no easy way to determine what version of the docs you are viewing without falling back regex hacks on the title or h1 element, which should probably not be relied on as a future-proof method, especially when localizing the docs. The URL itself also isn't reliable since these docs often live at vXX-latest. This meta tag provides a place you can reliably query the version of the node docs, whether as a script on the page (as in the other commit), or a program operating on the page (for example if you are running end-to-end tests with puppeteer), or even when visually comparing a diff.

[tolmasky]

@addaleax

Yes, CI automatically rebases, so nothing to do there – rebasing can be helpful if it resolves merge conflicts or it’s a very outdated PR, but otherwise, it can obscure whether new changes have been added that would need to be reviewed and tested separately or not – so no rebasing is definitely okay here.

Makes sense, thanks!