Improve ttfautohint hinting

[BladeMF]

I know it's not your job, but please can you guys help me identify what is missing in the patched NerdFonts since 2008 onwards so that the height changes so significantly - ryanoasis/nerd-fonts#519?

I really like to use the new versions and I really like oh-my-zsh, but I am stuck at the 2005 version.

I'm attaching patched fonts from the latest version (where the result is the same). static.zip

Edit: added version
Windows Terminal Preview
Version: 1.4.2652.0

[DHowett]

Hmm.. could it be the “use typographic metrics” flag?

[BladeMF]

Wow you're fast! Thank you very much for responding. If I can add the flag using, say, font forge, I would be so grateful! Especially if someone pointed me at some directions how.

[BladeMF]

Does this mean it is not?

[BladeMF]

Some more info (sorry for the spam):

Found 2 differences and changed them and re-exported the font, had no difference.

Also, Word does not render the fonts differently:

[aaronbell]

Forgive me if I say that the NerdFonts patcher is something like taking a sledgehammer to a nail. It does the job but does so without consideration for preserving the integrity of the original font. At least, that was my experience in looking at the resultant font file last year.

OK, so looking at the version you provided versus our latest version, I see some modifications that could result in spacing changes. For example, modification of the xMax and yMax values in the head table and an increase in ascender in the hhea table, but the changes that you're talking about seem to be inconsistent across OTF and TTF, and generally with glyphs appearing bigger than they should. Is that accurate?

That would lead me to think that the hinting may be responsible. If the hinting is different between the two fonts (and I'm not sure if Nerd Fonts is changing anything on that front), then x-heights can change, and glyphs may appear bigger.

Unfortunately, it isn't anything we can help you with because, as I said, it has a lot to do with how the patcher works. I'm hoping to release a (mostly) complete NF version here at some point which would solve the issue for you, but that's still in the works.

[aaronbell]

Something else to keep in mind. Once we started shipping the variable font, we no longer manually hinted the static instances. So there would be a change in the rendering between the pre-variable static version of Cascadia and the post-variable static version.

If you’re comparing with the older version then that might be a root cause too.

[BladeMF]

@aaronbell, I totally agree with you on your point, but I have to check if something could be done. My faith stems from several the fact that the 2005 version worked - so it didn't get as "destroyed". Also Word doesn't seem to see the difference and I was hoping it would be something small.

...seem to be inconsistent across OTF and TTF, and generally with glyphs appearing bigger than they should. Is that accurate?

Yes and the OTF appears even more distorted (look for the small letter "i" - the stem nearly touches the dot). Try alt-tabbing between the 2008 OTF version and the 2005 version and it becomes obvious.

Once we started shipping the variable font, we no longer manually hinted the static instances.

That came to my mind as well. That is why I installed the static TTF regular version to see how that looks and it didn't look distorted.

So my very humble request is this - if this is something that is fixable or things I could try, I'd be very grateful to be pointed in that direction as I know next to nothing about fonts. So, is there something like this?

[aaronbell]

Yikes, that is definitely a significant change. TBH, I'm not even sure where to point you to get started. I'll give it a think.

Could you do me a favor and package up the pre-modded, and modded versions for the 2005 / 2008 fonts that you're using? Thanks!

[BladeMF]

The change is totally in the "what the..." range. Thank you so much for looking into this!

This is everything that I've tried - CascadiaCode.zip. Only the 2005 patched version looks like the original font. 2008 and 2009 look practically the same. Also, the 2005 was the last variable width font that patched normally. 2008 and 2009 variable width fonts crash the patcher. 2008 I tried both TTF and OTF and the OTF looks worse.

I'd be happy to help further!

[BladeMF]

The command I use is this, but I've tried all the parameters and nothing, including --careful and --preserve-line-height (or something simalar sounding):

cd /mnt/d/Downloads/CascadiaCode-2008.25/ttf/static
fontforge -script ~/font-patcher -c CascadiaCode-Regular.ttf

[aaronbell]

OK, I've had a chance to take a look through these and to be honest, I don't see any differences between the NerdFont version and the official version that would be responsible for the differences that you're observing (I do see other significant changes between the two fonts, but they wouldn't result in the x-height modification).

As I read through your concerns, though, something became more clear. The 2005 version was the last version of the TTF that worked OK. That was also the last release before we switched to the variable font version, which means it was the last version to have manual hints.

The 2008 and 2009 releases were variable font releases and contain manual hints. Static instances were then provided as a 'backup' option for users for whom the variable fonts don't work, and therefore are just autohinted with ttfautohint.

I believe that the reason why the fonts look different than the 2005 version is because ttfautohint is choosing to set the hints differently than my manual hints (used in 2005, and on the variable font versions of 2008 / 2009). In running ttfautohint, I didn't do any manual adjustment to how it interprets the font—modifying this would likely cause improved results, but given they're provided purely as backup, improving ttfautohint is a low priority. On the plus side, if you want to do so (and figure out some good settings for us to use), the documentation is pretty extensive.

As for the OTFs, these are all autohinted with psautohint and have had the same settings throughout. OTFs on Windows tend to be a mixed bag and TBH I'm not as knowledgeable about the rasterizer interprets (or ignores) included ps hints. So can't comment on that.

[BladeMF]

Again, I appreciate you looking into it! I wonder why my test with installing the actual backup TTF didn't show distortions. Probably I didn't test it right. But if you are correct, the static TTF, if installed, will show the same distorted result, right?

So you are saying that I need to play with ttfautohint's settings and produce static fonts that look better, right? If so, I would appreciate a head start, maybe the settings you used? Maybe some hints? I am unsure if my knowledge will suffice for a useful test, but I am always willing to try stuff :)

[aaronbell]

Yes, the static instances that we provide should show the same effects on your device as the modified version. If you want to try that, I'd suggest uninstalling all of the versions, and installing one of the PL ones (to avoid overlap with one of the Windows Terminal fonts).

I used the standard setting with pretty much no modifications. Here's the current way we call it:
ttfautohint --stem-width nsn --reference "CascadiaCode-Regular.ttf" "CascadiaCode-Regular-hinted.ttf"

Looking at the documentation, I see something related to x-height which might do it.

[BladeMF]

Thank you, sound like fun!

Should I use the variable width font for base or the current TTF one and change its hints?

[aaronbell]

I'd suggest using one of the 2009 static TTF fonts. Thanks!

[BladeMF]

I'll write back when I have something. Thank you again!

[BladeMF]

Currently I can confirm that thet shipped 2009.22 static version appear simalarly distorted in height, so I will proceed with f@#kng around with ttfautohint and report back with successful settings (I hope) :-).

[BladeMF]

So @aaronbell , this is what I've found so far:

You definitely want --increase-x-height=0 (off ). Otherwise you can't stop the font becoming larger. By default, with these off, the capital letters and numbers appear 1px smaller (i.e. 9px height for 10px font):

You can also see how:

• the 0 is different (the dot is not in the middle)
• the hole in a is blurred
• The small m seems blurred

The a and m defects can probably be fixed using a different --stem-width-mode option.

Now the difference comes when you select a reference font. This is 2005.15 as reference. It is very different from the others - the capital letters and numbers are fine, but the small letters are all over the place:

This is 2007.01 as a reference. It looks the same as no reference. The same result is produced with 2008 and 2009 as a reference. I used the TTF variable-width fonts, but it doesn't seem to make a difference.

There is the base image:

Also, there is a slight difference when using --x-height-snapping-exceptions="-" (off) but leave -increase-x-height=14 - the height is the same as without it, but the position of the horizontal beam of B is 1px higher. Not sure what that means.

Anyway, I am out of ideas for now. The only way out I see is to have a reference font that corrects the capital letters. The ttfautohint docs seems to say the same thing.

What do you think? Does that help? Why does using 2005 and 2007 as a reference produce different results?

[aaronbell]

@BladeMF Thank you so much for taking the time to investigate this! And I'm sorry that it has taken me so long to follow up on this 😢.

From my understanding, the purpose of the reference font is to help ensure that different fonts within the same family behave in a similar way—so it is really important for the Bold to use the Regular as a reference font. I'm reminded that I think there was a change to the font metrics between version 2005 and 2007, so that might be why 2005 produces different results than 2007 and later.

So if we can't rely on the reference font to be able to change the results, what settings do you recommend to achieve the best rendering? It sounds like --increase-x-height=0 is the key change that will help keep the heights of the letters.

[BladeMF]

Hey @aaronbell , I am totally out of it now :-) The only thing we have to hope is that I documented my research well. It would seem that we want --increase-x-height=0 indeed.

[aaronbell]

Oh dear. Sorry it took me so long! I'll put that into the code and will hope for the best :x.

[Ririshi]

Sorry to revive a closed issue, but I think the issue still persists in v2102.003. Using Cascadia Code PL:

And the same window using CaskaydiaCovePL Nerd Font, patched with the --complete flag:

Here's a side-by-side comparison of some bits of those images:

It's especially visible in the powerline prompt.

[aaronbell]

Ah, that's unfortunate. I'll reopen this as it appears not to be resolved. However, customizing ttfautohint is not something I know, nor is it a priority for me at this time, so I can't promise a fix anytime soon.

Please let me know if you or someone you know would be able to assist :).

[BladeMF]

@Ririshi, I've opened an issue with Nerd Fonts as well. You can go over there and post and we can hope they respond.

[Ririshi]

Thanks @aaronbell, for reopening the issue.

Please let me know if you or someone you know would be able to assist :).

I am sad to say I know next to nothing about fonts. I was struggling to get the patching to work in the first place (I was trying to patch the variable ttf which isn't supported). I'll just keep my fingers crossed and hope someone else out there knows how to help :)

[Finii]

I stumbled about this issue in microsoft/terminal#14891 and started to investigate.
Starting with the release 2111.01 VF ttf version of Cascadia Code and create these static instances of Regular:

• without hinting
• with ttfautohint
• with Fontlab's TTH

Here are detail views of the changes, at 400% (no interpolation). The sequence starts and stops with the variable font.

without hinting

Horizontal bar of T looks really bad

with ttfoutohint

Capital height gets squashed

with TTH

Small letter e horizontal bar looks bad

From this I must say that the ttfautohint is the worst in my opinion, even no hinting is better. I will try to fine tune ttfautohint a bit.

[Finii]

Here the same steps as before but with font size 15, which seems the worst

Detail views of the changes, at 400% (no interpolation). The sequence starts and stops with the variable font.

without hinting

x height increases

with ttfautohint

x height increases

with TTH

x height stays the same, some other small shifts

[Finii]

Here we can compare the hints from Cascadia Cove VF (left) versus static (right):

This is opened in fontforge's "hinting simulator". It's a bit hard to see: Black outline is the actual outline in the font. Green outline is the outline after appying the instructions (hints). The squares are a renderer simulation. You see that the hints in the VF version make the green outline actually smaller, while the hints in the static versions make the outline quite taller.

Now that I have found a method to see this on Linux I can ditch the Windows and Mac machines and see if we can beat ttfautohint into submission ;-)

Edit: Hmm, the variable font hints also pull the dot a bit down (there is a green outline slightly lower), so that we get one very light grey pixel in the dot's bottom 🤔

[aaronbell]

@Finii I’ve got a separate question for you to discuss off-thread. Could I reach you via email?

[Finii]

@aaronbell Sure; my address is in every commit message, for example here (Signed-off-by).

Update on work done regarding the hinting: Tried a lot settings with ttfautohint to no avail. Now using a Windows 11 machine with all the software (ttfautohint, Fontlab, VTT, fontforge). While it is possible to get the x height smaller with ttfautohint (as found out already by BladeMF above) that costs us crispness of the x heights tops. That is in fact disabling the blue zone of the x heights. Which looks also not so good. With TTH it is no problem to autohint the static fonts and they come out very similar to the VF, but that is not a free solution. Then I found the vtt_data/ folder in this repo and wanted to try if they contain the original hinting data or at least the CVT, so that VTT could be used to hint the static fonts. That would even be possible in the/a release workflow.
Maybe ;-D

[aaronbell]

@Finii Great! Will do.

The one thing is that the static fonts don't have overlaps preserved, whereas the VF does. So the hinting data is a bit different. That said, the CVTs are definitely there, and could be used however you'd like :)

[Finii]

Here is the variable font opened in
a) Fontforge Linux b) VTT c) as rendered on my machine by Windows Terminal

The gamma looks differnt for each, but otherwise consistent. Note that FF gets confused by the overlapping outlines, it seems, making the marked pickel darker in the preview.

On Linux it is also rendered like this
a) as rendered on my Linux machine by Tilix b) VTT

This at least shows that it is consistent across different OS. And even the statics' x-height jumps up one pixel on Linux, where I would have sworn that it does not happen. I never noticed that.

---

Okay, now opening the release version of the static font in VTT, dropping program on opening:

x-height is 1 px too tall.

Then I just do a Tools -> AutoHint -> Light Latin Autohint:

And it looks ok. x-height as the VF, hinting not as advanced, but still...

Conclusion: I would drop ttfautohint and use just the dumb default AutoHint from VTT on the static files in the release workflow.

Well, than I have seen that the release workflow seems to run via Travis as Azure pipeline on MacOS.
I could not find VTT for Mac anymore, but maybe I'm just not good with searching.
Otherwise one could switch probably to Windows as pipeline OS, and/or add an additional step behind that just does the statics hinting on Windows.

Disclaimer: I checked just one glyph with one resolution... But it does look promising

---

Edit:

Ah, that's unfortunate. I'll reopen this as it appears not to be resolved. However, customizing ttfautohint is not something I know, nor is it a priority for me at this time, so I can't promise a fix anytime soon.

Ok, using VTT does not customize ttfautohint. If ttfautohint is a must I will have a look into its source code the next days. I must admit I did not find any knob to turn there either. Maybe we are out of luck with ttfouthint.

[aaronbell]

Thank you for your investigations into this!

The main reason why I was using ttfautohint is because it can autohint from a script / command line. AFAIK, VTT's autohinter has to be run from the application, which thus requires a manual step in the release process. As the static versions are (in my mind) secondary to the variable version, I wanted to simplify the hinting process on them.

But maybe it is worth just biting the bullet and running the autohinter on each static instance, storing the autohint code in the repro, and applying it to each static after generation. Should anything break we can re-run the autohinter manually to update the source as necessary. Either that, or convince MSFT to put the autohinter up along with the compiler...

Bit of bloat for the build system, but not overly complicated.

[Finii]

I did look a bit deeper into ttfautohint and I could not find any option to make the x-height blues a bit lower in general.

That leaves normal (manual) hinting. Just for experiments I ended up with these control instructions:

latn sinf @ a-z
# 'sinf' renders some lower cases smaller, marked with x
# Windows Terminal
#  x+++x+  +xxx++
o t 5-7,17-19 y -0.75 @ 6-50
n t 9-10,13-15 y -0.75 @ 6-50
d t 5-8,22 y -1 @ 6-50
s t 19-22 y -0.75 @ 6-50
s t 23-27 y -0.5 @ 6-50
e t 5-7,25-27 y -0.75 @ 6-50
a t 16-21 y -1 @ 6-50

Example command lines:

$ ./ttfautohint -v -m mycommandfile -F SINF7 ~/Downloads/Cascadia/ttf/static/CascadiaCode-Regular.ttf CCtest7.ttf
$ ftgrid -f 341 -r 96 15 CCtest7.ttf

The goal was to have a font where the string Windows Terminal has no visible x height change when switching between VF and static font at 15 pt. That would leave us with really manual hinting via delta hints, something not wanted I suppose. Forgive me my very broad and harsh hints.
But what was very unexpected is that the resulting font

• Renders equal x height on Linux ✔️
• Renders equal x height with fontforge's preview ✔️
• Renders equal x height with ftgrid ✔️
• Renders equal x height with VTT on Windows ✔️
• Does not render equal x height in Windows Terminal ??!! 🔴
  How can it be that VTT works and Terminal not 😭

At that point I stopped that approach. There is something going on that I do not understand. Hmm, I did not reboot, but used new Family Names for each test, so that should not be a problem.

So apart from patching the ttfautohint source and self-building in the workflow (this sounds hard but is in fact possible and I did it elsewhere in the past) I see no solution to utilize ttfautohoint.

Regarding VTT, I guess one can use it in the workflow, one could just send the needed keypresses to the GUI application and it will do what is needed. I can try to set that up if interested. I actually did not check if VTT can be operated just on keypresses, but I guess.

[longnguyen2004]

The one thing is that the static fonts don't have overlaps preserved

So, how about keeping the overlaps in the static fonts? After reading the code, it seems to be just a few boolean changes. That way, we can reuse the hints, and don't have to deal with ttf/psautohint. What's your opinion?

[aaronbell]

@longnguyen2004 Many applications assume that static fonts do not have overlaps, so preserving them, while it enables us to re-use the hints, can cause a multitude of rendering issues. As the static instances are being provided as "backup" for scenarios where the variable font is not viable, it doesn't make sense to introduce a different potential issue for the sake of hinting, IMO.

[vitorsr]

convince MSFT to put the autohinter up along with the compiler

Considering Adobe's impactful contributions to FreeType, and VTT being older than I am, I don't see that being impossible.

In microsoft/VisualTrueType#30 (comment), @paullinnerud said:

Removing the rasterizer and providing it in library/binary form has merit and is a possibility worth considering.

Plus, from what I gather, it is the GUI widget source that has non-Microsoft licensed copyrighted code, not the autohinter.

It then becomes a task of convincing someone to strip the GUI code and open source the rest of the library component.

No clue how VTT code itself is versioned internally but nowadays converting to Git and using git blame seems straightforward enough.

Unless, of course, the autohinter is tied to one or more particular renderer implementations. Then... who knows. Maybe gatekeeping it to Windows and replacing renderer calls with system calls to GDI, GDI+ and/or DirectWrite.

@PeterCon, @robmck-ms - apologies for the ping - would you know who could clarify? @mikedug, perhaps?

Microsoft Visual TrueType has today the only reliable variable font autohinter. It would have tremendous impact. (Second only to a DirectWrite autofitter and a Microsoft FreeType autofitter. I have a dream.)

[HinTak]

Removing the rasterizer and providing it in library/binary form has merit and is a possibility worth considering.

Plus, from what I gather, it is the GUI widget source that has non-Microsoft licensed copyrighted code, not the autohinter.

It then becomes a task of convincing someone to strip the GUI code and open source the rest of the library component.

...

No clue how VTT code itself is versioned internally but nowadays converting to Git and using git blame seems straightforward enough.

...

Unless, of course, the autohinter is tied to one or more particular renderer implementations. Then... who knows. Maybe gatekeeping it to Windows and replacing renderer calls with system calls to GDI, GDI+ and/or DirectWrite

...

No authoritative knowledge on this - MS folks please feel free to correct me - (1) the rasterizer was licensed from Apple in the 80's (almost 40 years ago) then heavily modified separately and went different directions by both Apple and Microsoft, (2) git itself is only about 20 years old... vtt is likely older than git, (3) the autohinter is indeed to some extent tied to particular renderer implementation.

The only/most interesting item is, of course, (3) . To what extent, and how important that might be, I have no idea.

I don't think the licensing of the GUI widget source is an issue; the practical usefulness of opening that part perhaps is, since GUI is necessarily windows only, and of no practical use on other platforms, except as inspiration / reference.

[vitorsr]

Thanks for the input, Hin-Tak.

Just to clear up some miscommunications on my part.

The rasterizer in the first case was referring to the rendering mode
simulations available in the GUI where the raster output is graphically
shown to the user. This can be achieved, for instance, via DWrite.dll calls
which have all required rendering modes. The headers are available in the
Windows SDK. This would unfortunately make VTT platform dependent.

In the second case, other uses of other rasterizers unbeknownst to us may
happen internally, however, and such a substitution is not possible if the
rasterizer is incompatible or if it makes use of functions not available on
the API.

On Git, the suggestion is that whatever version control software it may be
used, there is the possibility to convert to Git in order to use git-blame
and remove encumbered source code lines.

The overall theme is to comment that it is possible to locate encumbrances
and perform acceptable compromises in order to open-source the autohinter.

On patents, which was discussed on the quoted VTT issue, specifically in
connection to the autohinter, the Stamm patents, the Stamm-Hitchcock-Duggan
patents, and the Salesin-Wade-Zongker patent have expired.

The only encumbrance related to intellectual property here would be if the
autohinter is based on the Salesin-Wade-Zongker method, which is based on
transferring hints from a well-hinted source. In this case, the source and
its instructions are a possible area of copyright encumbrance.

[HinTak]

It is not about patents. The rasterizer was from Apple, and licensed to Microsoft, then heavily modified afterwards for 4 decades in GDI and DirectWrite. While Apple took the same code base in a different direction in their CoreText. I think it is highly unlikely that the original license agreement allows MS to open it up to 3rd party; and it is also highly unlikely that MS would go to the trouble of renegotiate that license to allow such opening.

Back to the 3rd point: autohinting of course depends, to some extent, the precise hinting behavior of the rasterizer(s). Auto-hinting "merely" inserts auto-generated hinting instructions (as opposed to "human-expert-edited/tuned" hinting instructions) to a font, and how they are processed, or ignored, is quite specific to the rasterizer(s). And to that end, despite sharing a distant ancestry from 40 years ago, font hinting on windows is quite different from font hinting on mac os now.