[INFRA] enable pandoc emojis for the pdf build

[sappelhoff]

The emojis for the contributors appendix are currently not rendered.

closes #551 by trying the fix from https://superuser.com/a/1383716/1191393

Applying "DejaVu Sans" may lead to correct rendering ... and that font should be included in the texlive docker --> https://tug.org/FontCatalogue/sansseriffonts.html but it's not working 🤷‍♂️ ... maybe https://ipfs-sec.stackexchange.cloudflare-ipfs.com/tex/A/question/234786.html as a fix?

done in this PR:

• using new font globally --> fixes emojis, doesn't break anything, looks nice
• updating a broken table in coordsystems appendix
• fixing broken dataset_description table
• adding new contribution types to emoji key in contributors appendix
• cleans up the PDF build script
• start using gfm instead of the deprecated github_markdown for rendering PDF reverted, because gfm does not seem to support the table wrap in the same way as markdown_github?
• for bids 2.0, link to https://github.com/bids-standard/bids-2-devel instead of old G doc
• enhance pandoc code to respect escaped table pipes: \|
• update contributors WIKI with new emojis: https://github.com/bids-standard/bids-specification/wiki/Recent-Contributors
• also close Link to bids-2-devel repo in issue template #566

[sappelhoff]

@Arshitha @sebastientourbier do you have an idea why it's not working for the emojis?

[sebastientourbier]

Seems pandoc is still using font [lmroman10-regular]...

[sebastientourbier]

On my side I tried to specify the DejaVu Sans font family by adding the following lines in listings_setup.tex:

\usepackage{fontspec}
\setmainfont{DejaVu Sans}

but it seems the UNICODE of these emoticons are still missing in DejaVu Sans.
For instance here is a line of the output I obtained:

[WARNING] Missing character: There is no 🤔 (U+1F914) in font DejaVu Sans/OT:script=latn;language=DF

[sappelhoff]

For instance here is a line of the output I obtained:

do you get an issue for all of these unicodes? or are now some of them supported at least?

[sebastientourbier]

@sappelhoff Managed to find a font (Symbola) that supports all emoji unicodes used by all contributors.
On my side (Ubuntu 18.04) I installed Symbola font with the following command:

sudo apt-get install -y fonts-symbola

I guess we could add something similar in the circleci job.

I also updated the listings_setup.tex as the following to use this new font:

% Contents of listings-setup.tex
\usepackage{xcolor}
\usepackage{graphicx}

\usepackage{fontspec}
\setmainfont{Symbola}

\lstset{
    basicstyle=\ttfamily,
    numbers=left,
    keywordstyle=\color[rgb]{0.13,0.29,0.53}\bfseries,
    stringstyle=\color[rgb]{0.31,0.60,0.02},
    commentstyle=\color[rgb]{0.56,0.35,0.01}\itshape,
    numberstyle=\footnotesize,
    stepnumber=1,
    numbersep=5pt,
    backgroundcolor=\color[RGB]{248,248,248},
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    tabsize=2,
    captionpos=b,
    breaklines=true,
    breakautoindent=true,
    escapeinside={\%*}{*)},
    linewidth=\textwidth,
    basewidth=0.5em
}

The emojis are now well rendered on my side.

[sebastientourbier]

For instance here is a line of the output I obtained:

do you get an issue for all of these unicodes? or are now some of them supported at least?

Not sure but quite a lot

[sappelhoff]

that sounds good!

I guess we could add something similar in the circleci job.

problem is we are using a docker container that does not have apt-get installed 🤔 we need a workaround such as

• install texlive in a different way
• build our own custom docker image

EDIT: Let me try using apt actually ... maybe I am wrong :-)

[sappelhoff]

nope :-/ we need a way to use the texlive docker image but at the same time include the new font. Probably a custom docker image is the way to go.

[sebastientourbier]

Seems apt-get is well installed based on https://app.circleci.com/pipelines/github/bids-standard/bids-specification/1444/workflows/035c86b7-2a75-41a5-a26b-923b730747aa/jobs/3530 but it cannot find the package. Maybe a apt-get update would solve it.
So what happens if you have the following command:

- run:
          name: install font
          command: apt-get update && apt-get install -y fonts-symbola 

[sebastientourbier]

@sappelhoff I could also give a try. I just would need the permission to push to this PR

[sebastientourbier]

Seems apt-get is well installed based on https://app.circleci.com/pipelines/github/bids-standard/bids-specification/1444/workflows/035c86b7-2a75-41a5-a26b-923b730747aa/jobs/3530 but it cannot find the package. Maybe a apt-get update would solve it.
So what happens if you have the following command:

- run:
          name: install font
          command: apt-get update && apt-get install -y fonts-symbola 

Seems to work

[sappelhoff]

thanks for trying out! I made the adjustments, let's see if everything works out in this PR now.

[sappelhoff]

awesome, that worked -- and the font even looks nice to me! Thanks for finding it @sebastientourbier :-)

I am going to add the new "all-contributors" kinds of contributions that we don't have yet, and then we can send this out to be reviewed by some other people.

we don't have the following emoji keys right now (see list). I'll check the box when I added them in this PR:

• userTesting
• security
• projectManagement
• platform
• mentoring
• maintenance
• data
• content
• business
• ally
• audio

Those that are not checked have not been added. Mostly because I can't think of an application of them within BIDS right now. Feel free to tell me of an example and we can add it!

[sappelhoff]

@sebastientourbier I rebased this PR on top of your enhancement of auto-table-fixing.

Please also see how I changed your str.split('|') to work with a regexp, so that escaped pipes: \| are ignored.

Now the interesting part --> since markdown_github is deprecated, I wanted to try to upgrade to the new gfm markdown style --> however, when doing that, the table cells did not wrap anymore! 😕

Do you know what's going on? You can easily check by comparing the CI outputs between

84c8261 and d33aef3

... simply scroll up to the commits, find the green check marks, click on them to bring up the list of CI checks, and from there you can see the CircleCI build printed output, as well as the artifact (PDF) itself.

I'd like to use gfm ... but if we cannot make it work with the new, nice tables then we'll have to stick with markdown_github for now.

edit, perhaps related:

• long text in pipe table does not wrap in pdf jgm/pandoc#4239

[sebastientourbier]

@sebastientourbier I rebased this PR on top of your enhancement of auto-table-fixing.

Please also see how I changed your str.split('|') to work with a regexp, so that escaped pipes: \| are ignored.

Now the interesting part --> since markdown_github is deprecated, I wanted to try to upgrade to the new gfm markdown style --> however, when doing that, the table cells did not wrap anymore! 😕

Do you know what's going on? You can easily check by comparing the CI outputs between

84c8261 and d33aef3

... simply scroll up to the commits, find the green check marks, click on them to bring up the list of CI checks, and from there you can see the CircleCI build printed output, as well as the artifact (PDF) itself.

I'd like to use gfm ... but if we cannot make it work with the new, nice tables then we'll have to stick with markdown_github for now.

edit, perhaps related:

* [jgm/pandoc#4239](https://github.com/jgm/pandoc/issues/4239)

It seems github_markdown is handling well pipe table , being able to span long content over multiple lines but it is not the case for gfm.
Perhaps related:

https://stackoverflow.com/questions/11700487/newline-in-markdown-table

It is suggested to use the html flag <br/> wherever the text content of a cell should break to a new line.
But, it is mentioned that the only valid option using pandoc to generate latex/PDF would be to adopt a grid table format. See https://stackoverflow.com/a/33052039 .

[sappelhoff]

merging, because spec content is untouched and we don't need excessive reviews for small infra enhancements.