Documentation LaTeX generation creates files that make pdflatex handle non-ASCII characters incorrectly

[huonw]

It seems our invocation of Pandoc on some files (e.g. guide-tasks.md) creates something equivalent to:

\documentclass{article}
\usepackage[utf8]{inputenc}
\begin{document}
π
\end{document}

which causes the following error

$ pdflatex unicode-input.tex
[...]
! Package inputenc Error: Unicode char \u8:π not set up for use with LaTeX.

See the inputenc package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              

l.4 π
[...]

The relevant section of (my) inputenc docs is:

1.3.3 Unicode char ⟨charcode⟩ not set up for use with LaTeX

This is the dreaded error message that one will receive if the document contains an
UTF-8 character that isn’t known to LaTeX. It is quite possible that the character
looks very unsuspicious and is rendered perfectly in the editor.

For example, when entering a Euro symbol from the keyboard one may receive
this error rather than a typeset symbol. But if this happens the reason is simply
that the document doesn’t load a font containing the Euro symbol, e.g., via the
textcomp package. Thus LaTeX does not know how to typeset one and therefore
responds with this error message.

However, even if LaTeX can type that character in question it may not
have been set up in which case you would need to do that yourself via
\DeclareUnicodeCharacter. If you provide these declarations for a full font en-
coding then please contribute that work to this package so that others can benefit
too.

[huonw]

I guess this might be a bug in pandoc... I don't really know. (The encoded character is U+03C0, i.e. it's not some peculiar pi glyph from a higher plane or anything like that.)

[huonw]

The above file works with both lualatex and xelatex i.e. it's pdflatex specific (which makes sense since the former two are designed for unicode input).

[pnkfelix]

cc me

[pnkfelix]

@huonw after reading some Q/A sites and finding the following links:

• http://tex.stackexchange.com/questions/13067/utf8x-vs-utf8-inputenc
• http://www.latex-community.org/forum/viewtopic.php?f=48&t=11965

I now have this variant of your file which does seem to render on my copy of pdflatex:

\documentclass{article}
\usepackage[LGRx,T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textgreek}
\begin{document}
π
\end{document}

[huonw]

I get the feeling that the comment

The best way to deal with unicode source (especially with non-latin scripts) is really XeLaTeX or LuaLaTeX.

on the first link is the correct solution: we live in a unicode world, so we might as well just use a unicode engine, rather than trying to bash pdflatex into submission. (Especially since we'd need to convince pandoc to output that code somehow.)

[pnkfelix]

@huonw I don't personally mind depending on XeLaTeX or some other unicode-enabled latex (as long as we select one that has wide support for our hosts, i.e. Mac OS X and Linux ... not sure if Windows/MinGW is as much as a priority as the other two).

[huonw]

XeTeX is bundled with TeX Live, MacTeX, and MiKTeX (see the History below for dates and versions). -- http://en.wikipedia.org/wiki/XeTeX

And some quick browsing suggests LuaTex is supported by those 3 distributions too (not 100% sure about MiKTeX), so it seems that platform support shouldn't be a problem.

[adrientetar]

TeXLive installs all major compilers indeed.

pandoc should use utf8 inputenc by default, see for yourself: default.latex; maybe the version on the bot is too old and doesn't have it. Also iirc latest pandoc parses unicode chars with \DeclareUnicodeCharacter but I'd need to check that up.

[pnkfelix]

@adrientetar the problem isn't whether utf8 inputenc is used; its that you need to load up other packages to get common non-ascii characters like π. (Compare the original test in the description to the version I found that one can build; both are using \usepackage[utf8]{inputenc}; the differences between them are the other usepackage commands that I had to add)

[adrientetar]

@pnkfelix We can use --include-in-header=header.tex to add packages. Or we can use lualatex by default.

[adrientetar]

BTW, are we really willing to make PDFs for guides? These just seem like quick things that you'd read up rapidly, not sure we need a LaTeX PDF for that.

[brson]

@adrientetar IMO we don't need PDFs at all, though I understand the desire to have a printable manual.

[adrientetar]

Fix would be to build with lualatex OR xelatex OR pdflatex (the opposite of what we’re doing now), code is here:
https://github.com/rust-lang/rust/blob/master/mk/docs.mk#L79

lualatex has been endorsed as pdflatex’s successor and ships with texlive since quite some time now.

[steveklabnik]

I think since we have the preferred latex stuff in the right order, this is effectively good enough.