Missing international characters in the bundled font

[miko]

For this adoc file I get my national letters stripped out of my name, because:
"Asciidoctor PDF uses custom fonts. When I created the font files that are bundled in the gem, I did not include all international characters in the font (something I've been meaning to fix)." (see issue #69 )

= Hello World
Michał Kołodziejczyk <miko@example.com>
1.0.0, 2014-12-10

Hello world!

[mojavelinux]

Thanks for providing the use case. We'll get this fixed up!

[goeh]

Is this the same problem that affects Swedish characters in source blocks? Or is that a different issue?

= Test Swedish characters
Göran Ehrsson <goran@example.com>
v0.1, 2014-12-12

This is a test document for asciidoctor-pdf and swedish characters (åäöÅÄÖ).

"Räksmörgås" is Swedish for *Shrimp Sandwich*

Räksmörgås enclosed in backticks `Räksmörgås` is not rendered correct by asciidoctor-pdf.

[source,html]
.SourceBlockFail.html
----
    <h1>Räksmörgås</h1>
----
Standard blocks renders ok
____
This räksmörgås is wonderful!
____

[mojavelinux]

Exactly the same issue.

The fonts that we bundle with Asciidoctor PDF should be complete fonts because Prawn already optimizes fonts when it "bakes" the PDF file. Thus, there's no reason for us to be pre-optimizing the fonts by throwing away characters as we are currently doing.

(I might still reduce the fonts slightly, but only to eliminate very rare languages, certainly nothing that would affect the Western sets).

[mojavelinux]

Thanks for the additional use case!

[kubamarchwicki]

I see the international characters working with Asciidoctor PDF alpha 8 - which is great progress! Woohoo!

{ ~ } master » asciidoctor-pdf -V
Asciidoctor PDF 1.5.0.alpha.8 using Asciidoctor 1.5.2 [http://asciidoctor.org]
Runtime Environment (ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]) (lc:UTF-8 fs:UTF-8 in:- ex:UTF-8)

However, possibility for custom fonts would be ideal. At the moment chars get rendered, but are not supported by the default font.

= Zażółć zółtą jaźń

Hello world!

gets rendered as follows:

[mojavelinux]

It's highly dependent on which characters. When I produced the current Noto Serif fonts, I included a broad range of characters, but not Cyrillic and the CJK characters are not part of that font. That's the change that's being requested.

[kubamarchwicki]

Got it. Noto is a custom font? I see two options. I'm happy to help with extending the font with Easter European - if possible.
Or a way to configure some other (less pretty fonts) - for the time being.

[mojavelinux]

I subsetted the font to make the gem smaller because at the time we had lots of different fonts. Now we have room to include the full font, so it's just a matter of swapping it in. I'll better document those steps when I do so it's clear what the situation is.

[kubamarchwicki]

👍

[mojavelinux]

I want to clarify that we still have to process the font because Prawn requires that we rewrite the kerning information to a separate table in the font.

[mojavelinux]

@goeh Actually this is slightly different because you're pointing out missing characters in the code font. This is separate from the body font. I intentionally subsetted the code font to only include ASCII to keep the size of the gem down. While the fallback font will catch these now, we still may want to reconsider that decision as code can have non-ASCII characters (comments, strings, etc). Let's tackle that in a separate issue.

[mojavelinux]

@goeh You will no longer be missing characters in monospaced text, but those characters will be coming from the fallback font atm.

[mojavelinux]

With the proposed pull request (#281), I think we're in a much better position to support a broad range of languages out of the box. We now include Cyrillic (for Russian, etc), Vietnamese characters and Latin Extended-A (for Polish, Czech, Turkish, etc) in the subsetted Noto Serif font. There really isn't much we are leaving out that the main Noto Serif font doesn't provide. The fallback font will catch Japanese characters as well as missing glyphs in literal text.

The only glaring void that remains is Chinese because we aren't yet bundling a font that has offers a complete set of glyphs (only those used in Japanese too). I think the right solution for that problem is a separate gem, which has already been setup.

Of course, we simply can't cover all languages with a single font...which is why I think the right long-term solution is to have gems for each language group that isn't covered by the fonts shipped in Asciidoctor PDF core.

[mojavelinux]

I'm going to document somewhere under docs the character ranges that are included in the bundled fonts. I'll also document why we subset / regenerate the fonts and how to do it for your own font.