Use Latexmk in a more efficient way for 'xelatex' as latex_engine

[jfbu]

Subject: improve (slightly) efficiency of xelatex builds

• Refactoring

We have said for a long time in our docs:

Also, if latexmk is at version 4.52b or higher (January 2017) LATEXMKOPTS="-xelatex" speeds up PDF builds via XeLateX in case of numerous graphics inclusions.

After some complex chase (as I could not find a Change log with release dates at Latexmk home going far back enough), it appears that from this Changes files without dates and CTAN announcements that both -xelatex and -lualatex options have been available since at least 2013, but that there was important changes at 4.52 of January 2017 to which the above tip was actually referring.

Starting from this date the -xelatex option triggered faster builds by Latexmk because it produces the PDF only once at very end of the multiple LaTeX passes. In the intermediate passes it produces .xdv files which is a bit faster process than producing PDF file.

What my PR does is to modernize our Makefile to use that -xelatex option (and -lualatex). Currently I can only test it on TeXLive 2019 LaTeX distribution not on earlier ones (and of course on current TL2024). I thus think that the change will not impact Sphinx users, even with very obsolete LaTeX installations. Due to this I have added it to 8.x milestone but hesitate about asking for inclusion in 8.0.0 (ping @AA-Turner). I will be able to test on antequated systems only in a few weeks, and depending on my schedule during transfers perhaps not until September.

Only the Makefile is changed, the latexmkrc Perl configuration appeared to require no change.

One should not expect miracles though in terms of performance improvement.

Commits will be added to update docs after opening the PR.

Relates

• Full build with PDF is taking more than 24h python/docsbuild-scripts#169
• Maybe use xelatex instead of pdflatex by default #4159

[jfbu]

Testing make -C _build/latex all-pdf after make latex with a build of Sphinx own docs configured to use xelatex on a TL2019 is a bit disappointing.

note that these builds are on a very old laptop and everyone else is going to observe faster build times.

With this PR (xeatex 2019):

real	1m23.364s
user	1m20.824s
sys	0m1.979s

Without (of course cleaning all auxiliaries from latex build first) (xelatex 2019)

real	1m22.613s
user	1m28.739s
sys	0m1.992s

WIth TeXLive 2024:

With this PR (xelatex 2024):

real	2m13.976s
user	2m9.313s
sys	0m3.925s

Without this PR (xelatex 2024):

real	2m15.046s
user	2m18.895s
sys	0m4.023s

Each build uses 3 LaTeX passes.

Observe the way LaTeX is slower nowadays...

I also observed some warnings which look inocuous

xdvipdfmx:warning: Creating ToUnicode CMap failed for "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Regular-400.otf"
xdvipdfmx:warning: Failed to load ToUnicode CMap for font "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Regular-400.otf"
xdvipdfmx:warning: Creating ToUnicode CMap failed for "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Solid-900.otf"
xdvipdfmx:warning: Failed to load ToUnicode CMap for font "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Solid-900.otf"

(anyhow when xelatex produces a PDF it does it via an intermediate .xdv which is then removed after xdvipdfmx, but perhaps then the warnings for some reason do not show). Anyway the return status was 0.

Now, it is expected that there will be gains when the PDF uses a lot of graphics inclusions, which is not the case with our own docs.

Not having experimented with that, I see no strong case to merge this PR in time for 8.0.0 and risk some downstream problems. If volunteers have time to try out this branch on their project and report...

If globally the gains are non-existent as above timings indicate, maybe better not to change anything at all.

In the meantime the advice seems to be to stick with some TeXLive 2019 based LaTeX rather than rush to modern one...

edit: I had been wondering if my TL2019 binary xetex was perhaps exceptionally fast due to having been built from source with compiler optimizing flags, but I have now used it with the TL2024 LaTeX macro layer and there is no significant difference with what the tL2024 binary does.

(xelatex with this PR, binary from 2019, LaTeX macro layer from 2024)

This is XeTeX, Version 3.14159265-2.6-0.999991 (TeX Live 2019) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(./sphinx.tex
LaTeX2e <2024-06-01> patch level 2
L3 programming layer <2024-05-27>
(./sphinxmanual.cls

....3 passes plus xdvipdfmx at the end...

real	2m16.333s
user	2m12.578s
sys	0m3.061s

[jfbu]

I added the do not merge tag. Waiting for feedback that merging this would actually make a significative difference in big projects if they have time to try it out.

The main lesson is that you should set-up your build environment to use a complete LaTeX distribution dating back to TeXLive 2019 or more ancient. Avoid newer LaTeX (we can see a 50% increase in build time from 2019 to 2024 regarding sphinx.pdf !!).

Sphinx uses very solid LaTeX packages most of them not having changed for a decade or more and is has very substantial internal LaTeX coding for code-blocks, tables, and admonitions (inclusive of recent #12508 supporting colourful admonitions). It does not especially benefit from being run on current LaTeX. It was better in the past, as everyone knows.

Build times with pdflatex not xelatex for our own Sphinx docs (still on a very old laptop). Timings do not include make latex but only the PDF build time after it (make -C <BUILDDIR> all-pdf).

TL 2019 (pdflatex):

real	0m54.658s
user	0m52.729s
sys	0m1.382s

TL 2024 (pdflatex):

real	1m11.519s
user	1m8.189s
sys	0m2.808s

as mentioned in my previous comment, I have a slight doubt about whether my TL2019 binaries were self-compiled with optimization flags for speed, but I do not think so, and my current understanding is that the decreased performance is due to additional overhead from recent LaTeX macro layer, not from the binaries.

EDIT: to dispel my doubts I used the 2019 pdftex binary together with the 2024 LaTeX distribution and here is the result: (pdflatex 2019 binary used with 2024 LaTeX macro layer and packages)

This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./sphinx.tex
LaTeX2e <2024-06-01> patch level 2
L3 programming layer <2024-05-27>
(./sphinxmanual.cls

....huge console output...

real	1m12.331s
user	1m10.027s
sys	0m1.918s

This shows that the faster speed with TeXLive 2019 is not due to some special binary build I did from sources with compiler flags for speed. It is due to the extra stuff loaded in the LaTeX macro layer and the extra processing it has added. (it is not from used LaTeX packages, they must have very little changed).

[jfbu]

To test if your project using latex_engine='xelatex' would benefit from this PR, it is not even needed to pull this branch.

1. execute make latex
2. cd to the latex build directory and therein do make clean
3. you can now do time make all-pdf and record the result
4. then do again make clean
5. and time make all-pdf LATEXMKOPTS="-xelatex"

The behavior at step 5 is what this PR makes default. instructions given for OSes supporting make and passing env vars to it on the command line like I did above.