Full biblatex support: somehow add biber engine to the backend

[pkgw]

It seems that some biblatex modes require the backend program biber as an alternative to bibtex. (Cf issue #33.) I don't know anything about it, but perhaps we should add it.

To gauge interest, everyone is invited to add a short comment on this issue if they use biber in their day-to-day TeX'ing.

[pkgw]

Here's biber. It's written in Perl. Hmmm.

[flying-sheep]

definitely. bibtex has no unicode support and is generally weird and flaky.

any modern LaTeX toolchain is LuaLaTeX/XeTeX + biber.

[mbollmann]

Just stumbled upon this project (wonderful idea!), but only recently switched to working with LuaLaTeX+biber (mainly because of Unicode support), so would definitely need this.

[jtojnar]

In the worst case, you could use the provided binaries (~20 MB).

[telemachus]

Just chiming in to say, yes please! I installed tectonic and I was very happy with the results overall. But sadly, my citations are not processed. They're simply left in bold after the build process. I would switch from latexmk to tectonic in a heartbeat if this was working.

If you decide not to include support for biblatex or for the time being, I'd suggest that if biblatex is present, the build should fail or output a clear warning that citations via biblatex aren't supported.

Thanks.

[pkgw]

OK, seems that this is a popular request. Anybody want to attempt to rewrite biber as a Rust crate? :-)

In the meantime I will consider pull requests that let Tectonic invoke the biber executable as in other TeX engines. This very much goes against the goal of reproducible, self-contained document compilations, but we should also be practical here.

[joostkremers]

I just found out about Tectonic and am tempted to give it a try, but this issue would really be a deal breaker for me.

But even more so, since you aim for a fully Unicode-enabled environment, you really cannot rely on BibTeX. BibTeX will fail in subtle ways when using Unicode: it may sort things incorrectly and you may end up with things like ä in your list of references. But since BibTeX doesn't produce any errors, you won't know unless you carefully examine the output.

goal of reproducible, self-contained document compilations

Do you have any ideas on how to deal with updates of packages? Updates fix bugs, which is good, but they may also wreak havoc on reproducibility, which is bad...

[pkgw]

@joostkremers Thanks; you make a good point — if we're going to say that we're Unicode-friendly, it matters to support it in .bib files as well as .tex.

As for package updates, Tectonic's "bundle" system tries to provide a framework in which you get both reproducibility and the capability to upgrade packages. This StackExchange comment elaborates a tiny bit more, but I need to write some real documentation that explains the concept in more detail.

[pkgw]

This might well be Google-able, but if anyone has a sample document using biblatex and biber in a nontrivial way, that they don't mind sharing, could you post a link to the files? I want to take a look at this and figure out how hard it would be to get the less-than-ideal "shell escape" solution working.

[jtojnar]

You can try one of the examples provided with fithesis. Though it uses some packages other than biblatex which might turn out to cause problems.

[joostkremers]

Well, perhaps this will help. Take the following .bib file:

@article{Haempke2002,
	author = {Hämpke, Johannes},
	title = {Irgendein Titel},
	journal = {Irgendein Journal},
	year = {2002},
	volume = {5},
	number = {6},
	pages = {7-8}
}

@article{Holbein2003,
	author = {Holbein, Frank},
	title = {Some Title},
	journal = {Some Journal},
	year = {2003},
	volume = {1},
	number = {2},
	pages = {3-4}
}

Note the author name Hämpke, which contains an umlaut.

Then use the following .tex file, which uses biblatex with BibTeX as backend:

\documentclass{article}

\usepackage[style=authoryear-comp,backend=bibtex]{biblatex}
\addbibresource{samp.bib}

\begin{document}

\textcite{Haempke2002, Holbein2003}

\printbibliography

\end{document}

I called the .bib file samp.bib. When you run this through xelatex --> bibtex --> xelatex, the result is this:

Note how in the list of references Hämpke is sorted after Holbein, even though according to German sorting rules (and indeed according to English sorting rules), their order should be reversed.

The biblatex manual says the following about using the BibTeX backend (I don't know what all of them really mean, TBH):

Key limitations of the BibTeX backend are:

• Sorting is global and is limited to Ascii ordering
• No re-encoding is possible and thus database entries must be in LICR form to work reliably
• The data model is fixed
• Cross-referencing is more limited and entry sets must be written into the .bib file
• Fixed memory capacity (using the --wolfgang option with bibtex8 is strongly recommended to minimize the likelihood of an issue here)

BTW, a quick test suggests that you may get better results using bibtex8 as backend. But I have no experience using biblatex / biber for documents that are not in English and have different sorting requirements (e.g., in Swedish, ä is a separate letter that is sorted after z), so I can't compare them to bibtex8.

[edwardabraham]

A bit late to the discussion. But yes please, biber. One reason we use it is that customising the style files is easier with biblatex than with bibtex.

[WtfJoke]

Is there any progress on this?

[pkgw]

@WtfJoke Unfortunately, no, there hasn't really been any change here. There are other areas where shell-escape support would make people's lives easier (example), though, so it's looking more and more like that will be a valuable feature to prioritize.

[Arunscape]

I ran into this issue as well, so for anyone else who needs a temporary solution, I pieced together a Makefile:

# Makefile

.PHONY: all

all: report.pdf

%.pdf: %.tex 
        tectonic --reruns 0 $<
        # Run biber if we find a .bcf file in the output
        if [ -f $(notdir $(<:.tex=.bcf)) ]; then \
                biber $(notdir $(<:.tex=)); \
        fi
        tectonic --reruns 0 $<
        rm $(basename $<).bbl $(basename $<).bcf $(basename $<).blg

edit: just wanted to say that I came across tectonic only last week, and I love what you've created here👏 💯

[rugk]

To gauge interest, everyone is invited to add a short comment on this issue if they use biber in their day-to-day TeX'ing.

Well… bibtex is just outdated legacy software. (no Unicode support, not flexible etc.) Just like the thing tectonic seeks to replace… 😉

So as a modern tool, tectonic should somehow certainly support that.

[koalp]

Hello, I came across this awesome project and also use biber.
I tried to use it along biber but have an incompatible version of biber installed (newer than needed by the packaged biblatex).

I will surely create a container image to be able to personally use and test this tool but don't think I will use it in makefiles or pipelines before biber is integrated.

[WtfJoke]

@koalp I experienced the same issue and created a docker image. You can find it here:
https://github.com/WtfJoke/tectonic-docker

However I didnt update the image in a while.

[RicardoMonteiroSimoes]

Could this be the reason for my failing Glossary/References print? We're rather new to LaTeX and trying to get things running, from what I've tried tectonic is by a very large margin the fastest compiler to use as action, but it doesn't generate glossary etc. I've tried the https://github.com/WtfJoke/tectonic-docker action with biber install, but that leads to a fail:

note: Writing `main.ist` (918 B)
note: Writing `main.glo` (1.18 KiB)
note: Writing `main-blx.bib` (343 B)
note: Writing `main.out` (3.34 KiB)
note: Writing `main.toc` (5.88 KiB)
note: Writing `main.acn` (233 B)
note: Writing `main.aux` (16.85 KiB)
note: Writing `main.bbl` (513 B)
note: Writing `main.pdf` (1.88 MiB)
note: Writing `main.run.xml` (2.60 KiB)
note: Writing `main.tdo` (2.69 KiB)
INFO - This is Biber 2.16
INFO - Logfile is 'main.blg'
ERROR - Cannot find 'main.bcf'!
INFO - ERRORS: 1

Not entirely sure if the error is on my side, but considering normal compilation on desktop with TeXStudio

[WtfJoke]

@RicardoMonteiroSimoes
Im not sure if that could be the problem...
Have you used tectonic-docker or the tectonic github action? Did you run tectonic using these commands?

1. tectonic --keep-intermediates --reruns 0 main.tex
2. biber main
3. tectonic main.tex

Do you have an example of your tex files?

[RicardoMonteiroSimoes]

@WtfJoke
I managed to fix it, someone in my team had set the backend of biblatex to bibtex, which explains why it didnt compile.

Yet my problem still remains about the glossary/references part. They don't want to compile and print in the action:

- name: Setup tectonic with biber
        uses: wtfjoke/setup-tectonic@v1
        with:
          github-token: ${{ secrets.AUTO_RELEASE_TOKEN }}
          biber-version: "latest"
          
      - name: Run Tectonic + Biber
        run: |
          tectonic --keep-intermediates --reruns 0 main.tex
          tectonic main.tex

Currently trying to see if we can make the repo public with my team, but here are the important snippets:

% !TeX program = lualatex

\documentclass[a4paper]{article}

% Packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage[backend=biber, style=ieee, citestyle=ieee]{biblatex}

% Document structure %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
	\sloppy
	\pagenumbering{roman}
	\color{content_normal}
	
	\begin{titlepage}
		\maketitle
		\thispagestyle{empty}
	\end{titlepage}

	%\subfile{sections/disclaimer}

	\tableofcontents
	\newpage
	
	\pagenumbering{arabic}

	
	%\subfile{sections/example}
	\subfile{sections/productvision/introduction}
	\subfile{sections/productvision/produktidee}
	\subfile{sections/productvision/spielregeln}
	\subfile{sections/productvision/umsetzung}
	\subfile{sections/productvision/risiken}
	\subfile{sections/productvision/architektur}
	\subfile{sections/productvision/technologien}
	\subfile{sections/productvision/miscellaneous}	
	%\subfile{sections/context}
	%\subfile{sections/objectives_functional_overview}
	%\subfile{sections/quality_attributes_non_functional_requirements}
	%\subfile{sections/constraints}
	%\subfile{sections/principles}
	%\subfile{sections/architecture}
	%\subfile{sections/external_interfaces}
	%\subfile{sections/code}
	%\subfile{sections/data}
	%\subfile{sections/infrastructure_architecture}
	%\subfile{sections/deployment}
	%\subfile{sections/operation_support}
	%\subfile{sections/decision_log}	% add sections here
	
	\newpage

	\printglossary[type=\acronymtype]
	\printglossary
	
	\newpage
	\printbibliography[heading=bibintoc]
	
	\newpage
	\listoftodos[Notes]
\end{document}

a local compilation works fine, so it shouldnt be a bad file

[WtfJoke]

FYI: I took your tex file and tried to make it compile (by making some changes) and it worked... you can check out https://github.com/WtfJoke/setup-tectonic-sample feel free to compare the files. Not sure if its an issue with the setup-tectonic action (feel free to open an issue as this goes a bit offtopic here)

[aterenin]

+1 to this issue, for many of the same reasons above, including Unicode support.

Is the main difficulty here that Biber is implemented in Perl, which makes it difficult to "build from source and glue together" compared to BibTeX? Looking at the BibTeX engine, all I see is a single C file. Is this why we are talking about shell escape support?

[aterenin]

Sorry to double post (my phone doesn't want to let me edit) but wanted to add to the above: if this is remotely doable for a Rust-fluent TeX-internals-novice, I'd be happy to make an attempt here with some support on what needs to be done.

I'd love to use Tectonic as the day-to-day build tool for writing my PhD thesis!

[pkgw]

@aterenin That's right, I'd say that it is essentially impossible to try to offer Biber as a built-in option in the way that Tectonic handles bibtex. The least-impractical approach would probably be an independent, work-alike Rust implementation. I'm not going to try to stop anyone from undertaking such a project but it's certainly not high on my priority list.

We are decently close to working shell-escape with #708 but it turns out to be a subtler problem than I first appreciated, and I believe that this is something where it's important to provide a really robust, carefully-designed solution. I think I see that solution, but I personally won't be able to spend any time on it for a little while, ironically because I'm working on a major proposal where I'm finding myself quite annoyed by the limitations of plain bibtex ...

[aterenin]

That's too bad - sounds not particularly encouraging. Biber is far too large to be reimplementable by one person, though if this someday did happen, that would be awesome, as it would finally become just as fast as BibTeX on very large bib files.

I did a brief Google search, and Perl supports interop with C, including the ability to instantiate a Perl interpreter from within C code, which together with Rust FFI means that in principle one can start a Perl interpreter from within Rust, thereby running Biber's code, perhaps an appropriately patched version thereof.

However, whether this has any advantages over just downloading a Biber binary and invoking it as a shell command - this is unclear.

[aterenin]

I wanted to add that perlcc was mentioned in the Biber issue plk/biber#338 as a way to get Biber to run on WebAssembly. Development made in pursuit of this goal could also make Biber+Tectonic integration more viable.

For the moment, I've been running Biber separately from VS Code by running tectonic --reruns 0 to generate the bcf file, then running Biber to generate the bbl file, and running tectonic to build the remaining document. This works great! It's even possible to create a hash of the bcf file to avoid re-running Biber if this file is unchanged, which can greatly improve compile times for documents with large bib files - this can be achieved with the one-liner if [ \"$(openssl md5 %DOCFILE%.bcf)\" != \"$(cat %DOCFILE%.bcf.hash)\" ]; then; biber --nolog %DOCFILE%; openssl md5 %DOCFILE%.bcf > %DOCFILE%.bcf.hash; fi.

This said, integration would be a step up, because this could all be theoretically done from within Tectonic, which could handle all of this automatically in addition to keeping Biber up-to-date in a reproducible manner.

[vadimkantorov]

I found perlcc / staticperl not reliable enough to compile all required packages. And an easier to reason/debug route may be as proposed by @aterenin:

1. Compile statically Perl and all dependencies, make sure they run biber on x64
2. Prepare a small C-based Perl embedded interpreter runner - make sure it works
3. Compile it all with Emscripten

I have advanced towards (1) in https://github.com/vadimkantorov/buildbiber/blob/master/emperl.sh, but the script is in some broken, unreadable shape for now

[pkgw]

OK, as of the 0.7 release, Tectonic now supports execution of biber as an external tool. The shell-escape work set up the technical pieces needed, and my overall feeling now is that it's better to probably compile a document correctly as opposed to definitely compile it incorrectly.

The current support doesn't let you pass command-line arguments to biber. It ought to be straightforward/nice to make that possible using the V2 Tectonic.toml scheme or a new -Z unstable option.

In principle I would certainly like to avoid having to invoke an external program, but as per the recent discussion, my guess is that it would be a pain to get that to work correctly.

[aterenin]

Amazing! 🎉🎉🎉

Unfortunately, I'm getting an error when using this.

aterenin@MacBook-Pro Thesis % tectonic --synctex thesis.tex
note: generating format "latex"
Running TeX ...
warning: thesis.tex:148: Overfull \hbox (16.34637pt too wide) in paragraph at lines 148--148
warning: thesis.tex:174: Overfull \hbox (18.73097pt too wide) in paragraph at lines 174--174
warning: thesis.tex:214: Overfull \hbox (13.0286pt too wide) in paragraph at lines 214--214
warning: thesis.tex:230: Overfull \hbox (4.78604pt too wide) in paragraph at lines 230--230
warning: thesis.tex:245: Overfull \hbox (20.94623pt too wide) in paragraph at lines 245--245
Running external tool biber ...
error: the external tool exited with an error code; its stdout was:

===============================================================================
INFO - This is Biber 2.16
INFO - Logfile is 'thesis.blg'
INFO - Reading 'thesis.bcf'
INFO - Found 5 citekeys in bib section 0
INFO - Processing section 0
INFO - Looking for bibtex format file 'citations.bib' for section 0
ERROR - Cannot find 'citations.bib'!
INFO - ERRORS: 1
===============================================================================
error: its stderr was:

===============================================================================
===============================================================================
error: the external tool exited with error code 2

[pkgw]

@aterenin Would you be able to file a separate issue with a minimal failing example? I was kind of surprised that my initial testing didn't run into problems such as yours ...

I'm sure that the issue here is related to the fact that the biber tool gets run in a temporary directory, because I very much want to make it possible for Tectonic to run with read-only document source directories. There's logic that tries to fill that directory with all of the files that biber (or other tools) will need, but it is evidently incomplete.

[clbarnes]

How would the recommended strategy above, i.e.

tectonic --keep-intermediates --reruns 0 main.tex
biber main
tectonic main.tex

work using the V2 commands/ layout used in tectonic's current docs, i.e.

src/
    _preamble.tex
    index.tex
    _postamble.tex
Tectonic.toml

as created by tectonic -X new my_project, and built with tectonic -X build?

[pkgw]

@clbarnes I'm afraid that functionality hasn't been wired in yet. My idea would be to add a keyword something like

early_helper_commands = ["biber"]

that would automate this sort of sequence. It should be fairly straightforward to implement with the existing infrastructure ... someone just needs to sit down and do it.

[s-weigand]

Both on windows and linux (both conda installation and biber on path) I'm getting this error

$ tectonic src/master_thesis.tex
Running TeX ...
Running external tool biber ...
error: not found
$  tectonic --version             
Tectonic 0.9.0

With

\usepackage[backend=biber,style=phys,mcite=true,defernumbers=true]{biblatex}

Has anyone encountered this error and knows a solution?

[pkgw]

@s-weigand On linux do you happen to be able to do an strace of your tectonic invocation and see what system call is causing the error? If you don't know what I'm talking about, don't worry about it.

[mseri]

Do you have biber installed? I got a similar error when I did not have it in the path, disappeared once I added it

[s-weigand]

Yeah I have it on the path, just the error message was a bit misleading 😅

The problem was that I'm currently just setting up the structure and reused the preamble from my bachelor thesis which included

\usepackage{notes2bib}

Thnx to the tip of @pkgw I ran it with strace (strace tectonic src/master_thesis.tex) and found this line

openat(AT_FDCWD, "src/notes2bib-master_thesis.bib", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

So it looks like notes2bib autogenerates a bib file, which it didn't in my case since I wasn't using it yet 🤦‍♀️
I think it would be helpful to know which file wasn't found

After solving this issue my next one shows a helpful error message 😅

Running external tool biber ...
error: the external tool exited with an error code; its stdout was:

===============================================================================
INFO - This is Biber 2.17
INFO - Logfile is 'master_thesis.blg'
INFO - Reading 'master_thesis.bcf'
ERROR - Error: Found biblatex control file version 3.7, expected version 3.8.
This means that your biber (2.17) and biblatex (3.16) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.
INFO - ERRORS: 1
===============================================================================
error: its stderr was:

===============================================================================
Use of uninitialized value in quotemeta at /usr/share/perl5/Biber/Config.pm line 228.
Use of uninitialized value $tool in concatenation (.) or string at /usr/share/perl5/Biber/Config.pm line 307.
===============================================================================
error: the external tool exited with error code 2

Guess I will have to manually copy a newer version of biblatex to the cache folder in the manifest there seems to be a mapping 🤔

cat .cache/Tectonic/manifests/264dfa0c090c395eb18859e35263a8f12b34678a3e01e2847690a5b3518f8360.txt | grep
biblatex
biblatex.sty 506356 fe3dbad513ad7177261658301020726ec37e8a5a8c8f0f2a7333276b8a38d797
biblatex.def 91946 6e143a287d28753c051af61ebb01524e9806454691432c1f7a0de9cb695caf10
biblatex.cfg 69 2f6e01efbb466a252120f07c8e12d38a34e747b01ba0b9ba2b7d66b4386bb5b4

Damn, why is it always so hard to get your latex env up and running? 😅

[pkgw]

This kind of stuff is why I'm always trying to minimize dependencies on external tools, sigh.

It's a little gross to overwrite files in the local cache but I presume that you have a sense of what you're getting yourself into if you do that! You should also be able to drop updated biblatex.sty and friends into your document source directory, or use the -Z search-path unstable option if you have lots of documents and don't want to copy the file around everywhere.

I think that when we update to TeXLive 2022, the built-in biblatex.sty will update to become compatible with your version of biber. I would like to do that update sooner rather than later, but unfortunately it's been very hard to find the time lately.

[Neved4]

ping @s-weigand your version of biber should work now, See: #893 (comment)

[s-weigand]

@mnrvwl Yeah it does 🎉
@pkgw Thanks for the new release 😄 ❤️

[pkgw]

OK, thanks to everyone who's participated in this issue over the years (plural!). There is still plenty to do here, but I think that I am going to close this issue in favor of #893, which captures the current flavor of the challenges a bit more accurately in my opinion. At this point I feel pretty well convinced that we're not going to be able to bundle biber in Tectonic itself, so the issue is more about dealing with the versioning of that external dependency than eliminating it.