New feature: BMFont support

[laqieer]

BMFont is a widely used font format in game engines such as Cocos, Unity, Ren'Py and so on. This pr is to support it. Tools, template, examples and documentation are included. All changes are backward compatible, so old projects needn't any change to build. It will make life easier to import fonts.
BTW It also comes with a variable width Chinese font with 3500 common Hanzi and Japanese font with 1945 common Kanji so that users can use them to print Chinese and Japanese even if they don't want to enable BMFont support. Those fonts are not included or linked unless manually specified, so it won't increase size by default if they are not used. It will solve #6 and #11, and corresonding part in FAQ is also updated.

[GValiente]

First of all, thanks a lot for your work. Butano really needs tools for importing standard font formats, and you have made a tool for a very popular font format. Great stuff.

However, I don't understand why it needs to be integrated into the main Butano Makefile (butano.mak) nor why Butano's bn::sprite_font class needs to allow UTF-8 character collisions.

As far as I understand, your tool generates a bn::sprite_font (.bmp and .h files) from a .bmfont file. To integrate that tool into your project's Makefile you don't need to modify Butano: the external_tool example shows how to do that.

And about allowing UTF-8 character collisions, if a collision happens Butano is not going to be able to determine which sprite to render for the involved characters, so the tool should generate a collision free font, instead of allowing collisions in bn::sprite_font.

[laqieer]

why it needs to be integrated into the main Butano Makefile (butano.mak)

If it is used as an external tool, each project needs to be released with the external tool to build, so developers needs to mantain many copy of the same scripts. If I update the conversion script, it is hard to update all copies of the script released with projects on the Internet, which makes version control impossible.
And each projects' Makefile needs to add the same line to call the same external tool, which is also a code duplication. Developers may want to call their own external tool developed for their own projects rather than add it to all projects.
Using git submodule is a solution, but it is impossible to force every developer to use it. Therefore I think it is best for butano to support it natively instead of an external tool, or at least an external folder in butano project to provide the standard font conversion.
It won't be executed if there is no fonts specified in the project Makefile, so it has no confluence on projects' builld without fonts. Therefore I think it is okay to include it in butano Makefile like other assets. Unlike other assets, it doesn't force the existence of the corresponding asset folders like graphics and audio. The project build process won't complain the lack of fonts folder, so the integration doesn't do any harm.

why Butano's bn::sprite_font class needs to allow UTF-8 character collisions

I have already explained it in the commit message of 93bca7e. It doesn't disable the check. It adds a config for user to decide the threshold of the character numbers to skip it to avoid project build process from hanging. The check function is just a simple nested loop, which is O(N^2). If there are 2000 characters to check, it needs to compute 5000 x 5000 = 25000000 times, which will take a really long time. And it will exceed the default max times limit for constexpr operations, and the compiler will complain it and quit with error:

common_cn_variable_16x16_sprite_font.h:20:62:   in 'constexpr' expansion of 'bn::sprite_font(bn::sprite_items::common_cn_variable_16x16, bn::span<const bn::string_view>(common_cn_variable_16x16_sprite_font_utf8_characters), bn::span<const signed char>(common_cn_variable_16x16_sprite_font_character_widths))'
C:/Users/laqie/Projects/butano/butano/include/bn_sprite_font.h:85:71:   in 'constexpr' expansion of '((bn::sprite_font*)this)->bn::sprite_font::sprite_font((* & item), (* & utf8_characters_ref), (* & character_widths_ref), 0)'
C:/Users/laqie/Projects/butano/butano/include/bn_sprite_font.h:121:9:   in 'constexpr' expansion of '((bn::sprite_font*)this)->bn::sprite_font::_duplicated_utf8_characters()'
C:/Users/laqie/Projects/butano/butano/include/bn_sprite_font.h:209:22:   in 'constexpr' expansion of 'bn::operator==((* & a), (* & b))'
C:/Users/laqie/Projects/butano/butano/include/bn_string_view.h:430:39: error: 'constexpr' evaluation operation count exceeds limit of 33554432 (use '-fconstexpr-ops-limit=' to increase the limit)
  430 |         const_pointer a_data = a.data();
      |                                       ^

Of course a user may really wants to do it for thousands of characters and is willing to wait several hours (maybe? idk) for it during project building. Then it is its free to set the config BN_CFG_SPRITE_TEXT_MAX_UTF8_CHARACTERS_FOR_DUPLICBN_CFG_SPRITE_TEXT_MAX_UTF8_CHARACTERS_FOR_DUPLIC to a big number like 5000, and it can be done thanks to d88f786.

the tool should generate a collision free font

The tools to make BMFont format fonts (*.fnt) mentioned in the documentation do it. They are already ensured to be collision free. It is not to allow collisions, but to provide users with a choice to skip the inefficient, time consuming and unnecessary double check on duplication for BMFont.

[GValiente]

why Butano's bn::sprite_font class needs to allow UTF-8 character collisions

I have already explained it in the commit message of 93bca7e. It doesn't disable the check. It adds a config for user to decide the threshold of the character numbers to skip it to avoid project build process from hanging. The check function is just a simple nested loop, which is O(N^2). If there are 2000 characters to check, it needs to compute 5000 x 5000 = 25000000 times, which will take a really long time. And it will exceed the default max times limit for constexpr operations, and the compiler will complain it and quit.

That's a problem that needs to be fixed instead of being hacked out.

Could you create a separate issue with a big bn::sprite_font (no bmfont) that exceeds compiler operation count limit? Thanks a lot.

[laqieer]

Could you create a separate issue with a big bn::sprite_font (no bmfont) that exceeds compiler operation count limit? Thanks a lot.

Done. #19.

[GValiente]

I hope with the new bn::sprite_font implementation you don't need modify Butano to convert BMFont fonts to the bn::sprite_font format.

About merging BMFont support into the master branch, I would prefer to have it as an external tool for now, and merging it in the future if it gains traction and becomes more popular after having matured a bit.

If you need more changes in the core for supporting BMFont or for anything else, please let me know :)

Again, thanks a lot for your help.

[laqieer]

Could you please advise me how to implement post-process on the generated graphics from fonts?
As shown above, the current solution is to modify $(GRAPHICSFILES) in butano's Makefile to add the dependancies. That variable is not accessible in project's Makefile. Notice that it doesn't work to add it to $(GRAPHICS).

[laqieer]

It also causes build problem to add generated graphics folder to $(GRAPHICS) in project's Makefile.
The current solution is to add it to parameter --graphics passed to butano_assets_tool.py in butano's Makefile.
3f636af
That parameter is not accessible in project's Makefile either.

[GValiente]

External tool is run before the rest, so if that tool generates files with known extensions (like *.h and *.bmp) in one of the folders specified in the user Makefile for each data type, you should not need to modify Butano.

Take a look at the external_tool example. In fact, you would only need to change GRAPHICS to support images generation from the external tool. Something like this should work:

TARGET      :=  $(notdir $(CURDIR))
BUILD       :=  build
LIBBUTANO   :=  /path/to/butano
PYTHON      :=  python
SOURCES     :=  src bmfont/src
INCLUDES    :=  include bmfont/include
DATA        :=
GRAPHICS    :=  graphics bmfont/graphics
AUDIO       :=  audio
ROMTITLE    :=  BMFONT
ROMCODE     :=  SBMF
USERFLAGS   :=  
USERLIBDIRS :=  
USERLIBS    :=  
USERBUILD   :=  bmfont
EXTTOOL     :=  @$(PYTHON) -B bmfont-script.py --build=$(USERBUILD)

The tool would generate output files in the bmfont folder that would be compiled later by butano.mak without modifications (I hope).

[laqieer]

Yes, you are right. I realized it later so I just marked them with strikeout. I need to add an independant cmd interface for butano_fonts_tool.py instead of a pure module for butano_asset_tool.py to import. Sorry to bother you. Thanks.

[laqieer]

New Feature

Auto trim useless characters in fonts to save space. 701e846.

Principle

1. counts characters in text files
2. extract counted characters from font files
3. insert the subset into the game

Backward Compatibility

It is disabled if no path to texts is set in Makefile. Old projects needn't any change to build without the new feature.

Example

• How to enable this feature
  laqieer/gba-free-fonts@aea8058
• Print mixing characters in various languages using one font
  laqieer/gba-free-fonts@4317096
  

hanamin_a.fnt
    ../../common/fonts/HanaMinA/hanamin_a.fnt font header written in build/hanamin_a_sprite_font.h (character number: 154)
    Processed character number: 154
    music_items_info file written in build/bn_music_items_info.h
    sound_items_info file written in build/bn_sound_items_info.h
    Processed audio size: 12 bytes
hanamin_a.bmp
    hanamin_a.bmp item header written in build/bn_sprite_items_hanamin_a.h (graphics size: 19744 bytes)
    Processed graphics size: 19744 bytes

Note: free font HanaMin A has 51,991 characters including CJK Unified Ideographs (URO, URO+, Ext.A) and CJK Compatibility Ideographs (with Supplement). It is difficult to import all of them into the game without the new feature. However, it is easy-to-use with auto-trim feature. The final game is only 155700 bytes with 154 used characters.

Test Result

• Built with auto-trim disabled

common_sc_variable_16x16.fnt
common_jp_variable_16x16.fnt
common_kr_variable_16x16.fnt
common_tc_variable_16x16.fnt
common_hc_variable_16x16.fnt
    ../../common/fonts/SimplifiedChinese/common_sc_variable_16x16.fnt font header written in build/common_sc_variable_16x16_sprite_font.h (character number: 3694)
    ../../common/fonts/Japanese/common_jp_variable_16x16.fnt font header written in build/common_jp_variable_16x16_sprite_font.h (character number: 2710)
    ../../common/fonts/Korean/common_kr_variable_16x16.fnt font header written in build/common_kr_variable_16x16_sprite_font.h (character number: 2453)
    ../../common/fonts/TraditionalChinese/common_tc_variable_16x16.fnt font header written in build/common_tc_variable_16x16_sprite_font.h (character number: 11254)
    ../../common/fonts/TraditionalChineseHK/common_hc_variable_16x16.fnt font header written in build/common_hc_variable_16x16_sprite_font.h (character number: 4918)
    Processed character number: 25029
    music_items_info file written in build/bn_music_items_info.h
    sound_items_info file written in build/bn_sound_items_info.h
    Processed audio size: 12 bytes
common_hc_variable_16x16.bmp
common_jp_variable_16x16.bmp
common_kr_variable_16x16.bmp
common_sc_variable_16x16.bmp
common_tc_variable_16x16.bmp
    common_hc_variable_16x16.bmp item header written in build/bn_sprite_items_common_hc_variable_16x16.h (graphics size: 3203744 bytes)
    common_jp_variable_16x16.bmp item header written in build/bn_sprite_items_common_jp_variable_16x16.h (graphics size: 819744 bytes)
    common_kr_variable_16x16.bmp item header written in build/bn_sprite_items_common_kr_variable_16x16.h (graphics size: 1133728 bytes)
    common_sc_variable_16x16.bmp item header written in build/bn_sprite_items_common_sc_variable_16x16.h (graphics size: 472864 bytes)
    common_tc_variable_16x16.bmp item header written in build/bn_sprite_items_common_tc_variable_16x16.h (graphics size: 2574240 bytes)
    Processed graphics size: 8204320 bytes

Built .gba file size: 8954348 bytes

• Build with auto-trim enabled

common_sc_variable_16x16.fnt
common_jp_variable_16x16.fnt
common_kr_variable_16x16.fnt
common_tc_variable_16x16.fnt
common_hc_variable_16x16.fnt
    ../../common/fonts/SimplifiedChinese/common_sc_variable_16x16.fnt font header written in build/common_sc_variable_16x16_sprite_font.h (character number: 115)
    ../../common/fonts/Japanese/common_jp_variable_16x16.fnt font header written in build/common_jp_variable_16x16_sprite_font.h (character number: 116)
    ../../common/fonts/Korean/common_kr_variable_16x16.fnt font header written in build/common_kr_variable_16x16_sprite_font.h (character number: 98)
    ../../common/fonts/TraditionalChinese/common_tc_variable_16x16.fnt font header written in build/common_tc_variable_16x16_sprite_font.h (character number: 120)
    ../../common/fonts/TraditionalChineseHK/common_hc_variable_16x16.fnt font header written in build/common_hc_variable_16x16_sprite_font.h (character number: 120)
    Processed character number: 569
    music_items_info file written in build/bn_music_items_info.h
    sound_items_info file written in build/bn_sound_items_info.h
    Processed audio size: 12 bytes
common_hc_variable_16x16.bmp
common_jp_variable_16x16.bmp
common_kr_variable_16x16.bmp
common_sc_variable_16x16.bmp
common_tc_variable_16x16.bmp
    common_hc_variable_16x16.bmp item header written in build/bn_sprite_items_common_hc_variable_16x16.h (graphics size: 72864 bytes)
    common_jp_variable_16x16.bmp item header written in build/bn_sprite_items_common_jp_variable_16x16.h (graphics size: 29600 bytes)
    common_kr_variable_16x16.bmp item header written in build/bn_sprite_items_common_kr_variable_16x16.h (graphics size: 42144 bytes)
    common_sc_variable_16x16.bmp item header written in build/bn_sprite_items_common_sc_variable_16x16.h (graphics size: 14752 bytes)
    common_tc_variable_16x16.bmp item header written in build/bn_sprite_items_common_tc_variable_16x16.h (graphics size: 57504 bytes)
    Processed graphics size: 216864 bytes

Built .gba file size: 354724 bytes

This feature reduces game size from 9MB (25000+ characters) to 0.35MB (569 characters) in this example.

[GValiente]

I'll close the pull request.

Feel free to create an issue if you need it.