Document how custom formats work / enable Pandoc templates

[allefeld]

As explained previously in #4889 (comment), I'm trying to translate my long-used Pandoc formats to Quarto. During my first attempt to do so, I stumbled upon what might be considered a bug in quarto create extension, but as pointed out in a helpful dev's comment, it can be easily worked around if one knows more about how custom formats work internally.

Now I ran into another issue: Apparently, embed-resources is incompatible with MathJax. The workaround provided there involves customizing the Pandoc template for HTML documents. In Pandoc, doing so is relatively easy: One uses --print-default-template=html, saves the output to a file, edits it, and provides it through template:.

Specifying a template is also possible in Quarto, but there seems to be no way of getting the default template. I did some reverse engineering, and discovered that before calling Pandoc, a file template.patched is created, which includes several (also created) sub-templates, and also this part:

$if(math)$
$if(mathjax)$
  <script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
$endif$
  
  <script>
    (function () {
      var script = document.createElement("script");
      script.type = "text/javascript";
      script.src  = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
      document.getElementsByTagName("head")[0].appendChild(script);
    })();
  </script>

$endif$

So, would it work if I copied this template.patched file, and made the addition script.data-external = "1";? Or is the template truly "patched", i.e. dynamically generated?

What I ended up doing is adding this code to my custom format:

embed-resources: true
# Hackery to make MathJax work with embedded resources
# see https://github.com/jgm/pandoc/issues/8362
# – make Pandoc embed math in MathJax format
# – neutralize Pandoc's script element, because it doesn't work with embedding
# – explicitly include a script element which is not included in embedding
html-math-method:
  method: mathjax
  url: ""       
header-includes: <script data-external="1" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" type="text/javascript"></script>

But this is a hack, and it would be so much easier if Quarto would provide a default template.

Speaking more generally: The power of Pandoc comes not only from the many output formats it supports and its many options, but just as much from its template engine. Quarto adds to this power by enabling computationally created content, but in its current form it takes away the easy templating.

Please document the way custom formats work internally, and in particular preserve Pandoc's awesome templating.

[allefeld]

It appears the issue with MathJax embedding went away, see jgm/pandoc#8362 (comment). I still wish Quarto would do less intransparent things, to make it easier for Pandoc users to adapt it to one's needs.

[allefeld]

I did some further reverse engineering, based on what happens with minimal: true and theme: pandoc, which seems to be the simplest setup:

Quarto calls its internal copy of Pandoc with the arguments

+RTS -K512m -RTS
--defaults .../quarto-defaults########.yml
.../quarto-input########.md
--metadata-file .../quarto-metadata########.yml
--data-dir /opt/quarto/share/pandoc/datadir

where ... refers to randomly named temporary directories, and ######## to a random identifier.

The defaults file quarto-defaults########.yml refers to a template file template.patched, which is created in a temporary directory. It includes partial templates metadata.html, styles.html, title-block.html, title-metadata.html, and toc.html, which are created in the same directory. Moreover, styles.html includes a partial template styles.citations.html, which however does not seem to be created in the same directory.

The files metadata.html, styles.html, and toc.html appear to be identically copied from a Quarto installation directory /opt/quarto/share/formats/html/pandoc. template.patched is a modified copy of template.html, where Pandoc's $math$ template macro is replaced by JavaScript code to load MathJax dynamically. A file styles.citations.html is present in that same directory (but is not copied).

On the other hand, title-block.html and title-metadata.html appear to be identically copied from yet another directory, /opt/quarto/share/formats/html/templates/.

The main template, template.patched, can be overridden with the template: option, but the partial templates cannot. Moreover, additional partial templates in the same directory as the specified main template are not copied to Quarto's temporary directory.

---

I assume there is a reason for this complexity, but it makes it extremely complicated to use one's own template.

What I'm trying to achieve is to create a custom format which supports all features of Quarto, or at least a large selection of those I might eventually use, but which is as lean as possible: No unnecessary JavaScript, no duplicate or mutually overriding CSS styles.

[allefeld]

Thanks for your detailed reply! Your future plans sound like what I wish for. I think customizable templating is important – your templates are nice but can't cover all use cases. Also, I like to control things. ;)

I wasn't aware of template-partials, thanks. Even after years of using Pandoc and going over the manual again and again, I still find new stuff – it's a very complex software. In the Quarto documentation I probably didn't see it because it's under "article templates"?

---

I currently have the following plan, and it would be great if
(1) you could let me know if you think this will work to get "a custom format which supports all features of Quarto":

• Write a Quarto test document with all the features I'd possibly want to use, and process it with

minimal: true
theme: pandoc
document-css: false

• Patch together a template from the different temporary files, in the hope that this is equivalent to what Quarto does internally.

• Modify this template to my needs.

However, chasing after temporary files is tricky, so it would be great if you could also answer this question:
(2) Is there any place in the Quarto installation which contains all templates and partial templates used by Quarto? Specifically for HTML, I found /opt/quarto/share/formats/html/templates/ and /opt/quarto/share/formats/html/pandoc/, and it is unclear to me how they relate to each other. Or should I use quarto pandoc --print-default-data-file templates/default.html5 and then the same for all used partials?

For the moment I'm focused on HTML, but in the future I'll probably try the same for LaTeX and beamer.

[allenmanning]

In the Quarto documentation I probably didn't see it because it's under "article templates"?

@cwickham I'm not sure if there is a place we can cross-link this, but I thought I would signal what seems to be a find-ability issue.

[cwickham]

Tracking the documentation issue at #5194

[allefeld]

In now read https://quarto.org/docs/journals/templates.html#template-partials, and it is close to what I was looking for with this feature request. I think it would be great if this material could be moved to a more visible place as @allenmanning proposed, and @cwickham in #5194.

But I still have some questions which aren't answered there:

• The Pandoc templates used by Quarto are in the repository in subdirectories of src/resources/formats, and specifically the main and partial templates for html under src/resources/formats/html/pandoc?

• What are the other directories under src/resources/formats/html/ for, in particular templates/?

• Is the repository directory src/resources/formats identically copied to /opt/quarto/share/formats/, i.e. can I take the files from there instead of from the repository?

• If I create a custom format e.g. for HTML, and as a starting point I want it to be identical to the default Quarto template, then I can use

  template: template.html
  template-partials:
    - metadata.html
    - styles.html
    - title-block.html
    - toc.html
    - styles.citations.html

  I can then modify these templates, and if I decide not to modify some of the partials, I can omit them from template-partials: because the original version is used instead – correct?

• I can instead use the complete template without partials, template: html.template – correct?

• But where do citations.styles and html.styles come in, they do not seem to be referenced anywhere? citations.styles is identical to styles.citations.html, so why duplicate it? html.styles is very similar to styles.html but not identical. Can I simply ignore these two files?

In summary, I now see that there is documentation though in a non-obvious place, but there are still some obscure unexplained aspects, and it would be great if in addition to making the documentation more visible, you could fill in the missing details.

[cderv]

@dragonstyle will be able to add to this as he know more about this but here is some answers

The Pandoc templates used by Quarto are in the repository in subdirectories of src/resources/formats, and specifically the main and partial templates for html under src/resources/formats/html/pandoc

Yes they are in the pandoc/ subdir you can found in some for the format directory in src/resources/formats

What are the other directories under src/resources/formats/html/ for, in particular templates/?

I believe they are our own partial templates injected as partial for pandoc in our own Quarto processing (whereas the templates partials in pandoc/ folder are derived from main pandoc tempalte - we split the default pandoc template). @dragonstyle Do you confirm ?

Example of usage at documentTitlePartial() function used to add some bootstrap stuff I believe

if I decide not to modify some of the partials, I can omit them from template-partials: because the original version is used instead – correct?

Yes the default template will be used

I can instead use the complete template without partials, template: html.template – correct?

This would be the default pandoc template yes. Without quarto specific in it. You would probably some quarto features though as there is no partials in this file and we use partials to inject some content too (as example above). But this is usual limitation when using replace template, or deactivate theming, etc... We build the partials system to allow tweating template more easily.

The main template used by Quarto which contains the partial call is template.html

But where do citations.styles and html.styles come in, they do not seem to be referenced anywhere? citations.styles is identical to styles.citations.html, so why duplicate it? html.styles is very similar to styles.html but not identical. Can I simply ignore these two files?

They are copies of the pandoc one that we retrieve when we update pandoc. We copy them for saving the states of what we download I guess - I think this is quite new and mayb not be necessary but it is possible that we modify them from there default in pandoc - hence the new name

Overall, to modify a template you just

• Modify one or several of the partials used in template.html - this allow you to tweak a small part of the whole template
• Replace template completely
• Use the include statement available to pass content as variable https://quarto.org/docs/output-formats/html-basics.html#includes