Windows binary distribution

[simoncozens]

We need one. But I don't have the resources to make one.

To make this happen we will need:

• A reproducible (hopefully scriptable) Windows build. See also Building SILE on MinGW32/Windows8.1 #82. We just made this harder by requiring ICU.
• A C wrapper around the SILE Lua script, so it can be more easily run as an EXE. (And without the Lua runtime.)
• To isolate the libraries which are required, so that they can all be shipped as part of a Windows distribution
• To ship all the needed files as a distribution and test them on a virgin system ie with no ming32/msys2 installed.

[indiamcq]

Simon,

I am working in mingw64 on Win 10. I have used your build from source instructions in this environment. I am thinking that this will create a sile.exe file, but as I don't know much about this stuff, I may be off track.

The ./configure is getting stuck on:

checking for FONTCONFIG... yes
checking for lua5.3... /mingw64/bin/lua5.3
checking if /mingw64/bin/lua5.3 is a Lua interpreter... no
configure: error: not a Lua interpreter

I had modified the configure.ac
from AX_PROG_LUA([5.1]) to AX_PROG_LUA

Also I modified the ax_lua.m4 to

  m4_define_default([_AX_LUA_INTERPRETER_LIST],
    [lua5.3 lua53 lua5.2 lua52 lua5.1 lua51 lua50 lua])

i.e it now includes lua5.3 that was missing. Though moving lua to the end may be wrong.

Any insights on what may be wrong? If what I am doing is off track then just say so.

Ian

[simoncozens]

I've no idea why that's failing. The test it should be performing is:

$ lua5.3 -e "print('Hello ' .. _VERSION .. '!')"
Hello Lua 5.3!

Can you attach the output of config.log?

[simoncozens]

Well, I just managed to replicate that error! Looking into it now.

[simoncozens]

It's this: https://github.com/Alexpux/MINGW-packages/issues/949
Not sure how to fix it but I think the end goal for Windows is going to involve a custom SILE binary anyway, so I wonder if we could actually get rid of this test.

[simoncozens]

OK, here's the state of SILE building on Windows so far:

• I'm building a Lua from source on cygwin/msys2/mingw64, using make mingw. It seems to work.
• I can't build the various extensions (lpeg, lxp, etc.) from luarocks, because the linker fails with all of the symbols from Lua being undefined.
• I can build them by hand if I add -shared /path/to/lua5.X.dll to the linker line. I don't actually know if this is the right thing to do or not.
• They work fine if the produced lpeg.so (.so not .dll because it's mingw) etc. is in the current working directory. If I put them somewhere in Lua's cpath, I get something like "required module not found" when I try to require them. This is obviously a problem.
• mingw's zlib doesn't have compress2, so libtexpdf can't build automatically. Spiking the check and building the library by hand seems to work OK.
• But then building justenough... is not working, as libtool is struggling to produce a DLL which links to another DLL. We just did this by hand for the other Lua extensions, so it must be possible, but libtool is complaining.

This is all happening in the Appveyor branch, with build logs here: https://ci.appveyor.com/project/simoncozens/sile/build/1.0.73

If anyone who knows Lua-on-Windows (I'm looking at you @pkulchenko) has time to have a look over what I'm doing and sanity check it, particularly appveyor.yml and .appveyor/build.sh, it would be very helpful.

[pkulchenko]

@simoncozens, I won't be able to help with appveyor config, but I can definitely try to build SILE from scratch on Windows, similar to what I did in #82.

A C wrapper around the SILE Lua script, so it can be more easily run as an EXE. (And without the Lua runtime.)

Why would you need this?

[simoncozens]

Thanks! Would be very handy to compare notes.

The idea behind a C wrapper (which I've now written) ties into my end goal for distribution on Windows: rather than installing Cygwin and Lua and Luarocks and everything else and then SILE, the installer just unpacks a punch of stuff into C:\Program Files\SILE, where all the dependency libraries - Mingw, Freetype, Harfbuzz, ICU, Lua and so on - are stashed away and nobody needs to care that they're running Lua and half of Unix when they type "sile foo.sil".

[pkulchenko]

Simon, I've made some progress on getting this to run, but on a snapshot from last year, which predates lfs and justenoughicu requirements.

I got it working using DLLs from mingw32/msys2 distribution; here is the full list of libraries used:

libbz2-1.dll
libexpat-1.dll
libfontconfig-1.dll
libfreetype-6.dll
libgcc_s_dw2-1.dll
libglib-2.0-0.dll
libharfbuzz-0.dll
libiconv-2.dll
libintl-8.dll
libpng16-16.dll
libtexpdf-0.dll
libwinpthread-1.dll
zlib1.dll
lpeg.dll
lxp.dll

The last two are Lua libraries and the rest are DLLs from msys\mingw32\bin folder. I also have justenoughharfbuzz.dll and justenoughlibtexpdf.dll I built using SILE scripts.

There have been couple of issues; first of all, I copied all the DLLs into the local folder from which I run SILE; if this is not done, then there is a high likelihood that one of the libraries will be present in one of the folders in PATH that sits before mingw folders, which is likely to make it fail, simply because it will be a 64bit library (from Ruby23-x64) or a library from a different version of mingw or something similar.

The second issue was with libtexpdf from mingw, which required fonts.conf file (I didn't use libtexpdf from SILE as I ran into issues with compiling it; maybe need to try one more time). It just couldn't find any of the fonts without fonts.conf being present. I fixed this by adding set FONTCONFIG_FILE=D:\full\path\to\sile\fonts.conf before launching SILE with the content of the file taken from shoes/shoes3#46 (comment).

This worked fine, but it took a while to complete the first run (probably about 20s) while the fonts were scanned and the cache was compiled. The subsequent runs were without much of a delay.

The next step would be to build a more recent version with justenoughicu and other additions.

[pkulchenko]

Couple of issues I ran into compiling the current master with msys2/mingw32:

  CC       justenoughfontconfig_la-justenoughfontconfig.lo
In file included from justenoughfontconfig.c:10:0:
silewin32.h: In function 'strcasestr':
silewin32.h:9:5: warning: implicit declaration of function 'tolower' [-Wimplicit-function-declaration]
     if (tolower(haystack[i]) != tolower(needle[matchamt]))
     ^
  CCLD     justenoughfontconfig.la
  CC       fontmetrics_la-fontmetrics.lo
  CCLD     fontmetrics.la
  CC       svg_la-svg.lo
  CCLD     svg.la
  CC       justenoughicu_la-justenoughicu.lo
justenoughicu.c:5:29: fatal error: unicode/ustring.h: No such file or directory
 #include <unicode/ustring.h>
                             ^
compilation terminated.
Makefile:653: recipe for target 'justenoughicu_la-justenoughicu.lo' failed
make[1]: *** [justenoughicu_la-justenoughicu.lo] Error 1
make[1]: Leaving directory '/home/Paul/sile/src'
Makefile:510: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

I fixed this by adding: -I/usr/include to DEFAULT_INCLUDES = -I. -I/usr/include.

  CC       justenoughicu_la-justenoughicu.lo
  CCLD     justenoughicu.la

*** Warning: linker path does not have real file for library -licuio.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicuio but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licui18n.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicui18n but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licuuc.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicuuc but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licudata.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicudata but no candidates were found. (...for file magic test)

*** Warning: libtool could not satisfy all declared inter-library
*** dependencies of module justenoughicu.  Therefore, libtool will create
*** a static module, that should work as long as the dlopening
*** application is linked with the -dlopen flag.
cp .libs/justenoughharfbuzz.dll .libs/justenoughfontconfig.dll  .libs/justenoughlibtexpdf.dll .libs/fontmetrics.dll .libs/svg.dll .libs/justenoughicu.dll  ../core
cp: cannot stat ‘.libs/justenoughicu.dll’: No such file or directory
Makefile:935: recipe for target 'all-local' failed
make[1]: *** [all-local] Error 1

This is fixed by installing a proper icu module: pacman -S mingw-w64-i686-icu; this still didn't quite work as it did find some libraries, but not others:

  CC       justenoughicu_la-justenoughicu.lo
  CCLD     justenoughicu.la

*** Warning: linker path does not have real file for library -licui18n.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicui18n and none of the candidates passed a file format test
*** using a file magic. Last file checked: D:/utils/msys64/mingw32/lib/libicuio.dll.a

*** Warning: linker path does not have real file for library -licudata.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicudata and none of the candidates passed a file format test
*** using a file magic. Last file checked: D:/utils/msys64/mingw32/lib/libicuuc.dll.a

*** Warning: libtool could not satisfy all declared inter-library
*** dependencies of module justenoughicu.  Therefore, libtool will create
*** a static module, that should work as long as the dlopening
*** application is linked with the -dlopen flag.
cp .libs/justenoughharfbuzz.dll .libs/justenoughfontconfig.dll  .libs/justenoughlibtexpdf.dll .libs/fontmetrics.dll .libs/svg.dll .libs/justenoughicu.dll  ../core
cp: cannot stat ‘.libs/justenoughicu.dll’: No such file or directory
Makefile:935: recipe for target 'all-local' failed

I fixed this by renaming ICU libraries according to this discussion: ICU_LIBS = -L/usr/lib -licuio -licuin -licuuc -licudt -lpthread -lm.

I also got message ./libtool: line 7916: func_munge_path_list: command not found, but it didn't seem to impact anything.

[pkulchenko]

Here is the list of additional libraries that is needed to decouple the installation from mingw:

libicudt57.dll
libicuin57.dll
libicuuc57.dll
libstdc++-6.dll
libzlib.dll (Lua library)

@simoncozens, I got it all running without issues, but the resulting PDFs have boxes instead of characters. What would be the reason for that? The console output looks correct:

luajit sile -e "require('core/libtexpdf-output')" -d fonts examples\simple.sil
This is SILE 0.9.5-unreleased
<examples\simple.sil>
[fonts] Looking for Gentium Plus;10;400;;normal;;LTR
[fonts] Resolved font family Gentium Plus -> C:/Windows/Fonts/GentiumPlus-R.ttf
[1]

[pkulchenko]

I rolled back changes to the earlier commit that works for me (e9d821d), but still using the new binaries and with justenoughharfbuzz.dll from the same commit (as I couldn't use the new one as the API has changed) everything works, so it seems to be related to the changes in harfbuzz.

[pkulchenko]

This is what SILE.shaper.debugVersions() shows:

Harfbuzz version: 0.9.40
Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
C:/Windows/Fonts/GentiumPlus-I.ttf:0    Unknown version
C:/Windows/Fonts/GentiumPlus-R.ttf:0    Unknown version

[pkulchenko]

@simoncozens, @alerque, any ideas on why I'm getting boxes instead of characters in the output?

[simoncozens]

OK, that's weird. Strange that it can't get a version string out of the fonts, which makes me suspect that either the fonts are bad or that the opentype parser isn't working on Windows. That shouldn't affect output, however. Could you try:

% ./sile -e 'require("core/debug-output")' examples/simple.sil

This will tell me whether or not the problem is to do with finding the right glyphs in the font (Harfbuzz problem) or outputting the glyph to pdf (libtexpdf problem).

[pkulchenko]

@simoncozens, here is the output:

luajit sile -e "require('core/debug-output')" examples/simple.sil 
<examples/simple.sil>
Set paper size 	365.6692953	561.2598486
Begin page
Mx 	50.3506
My 	65.1061
Set font 	Gentium Plus;10;400;;normal;;LTR
T	0	(I)
Mx 	61.3644
T	0 0 0 0	(have)
Mx 	87.3783
T	0 0 0 0 0 0	(seldom)
Mx 	123.3922
T	0 0 0 0 0	(heard)
Mx 	154.4061
T	0 0 0	(him)
Mx 	175.4200
T	0 0 0 0 0 0 0	(mention)
Mx 	216.4339
T	0 0 0	(her)
Mx 	237.4478
T	0 0 0 0 0	(under)
Mx 	268.4617
T	0 0 0	(any)
Mx 	289.4756
T	0 0 0 0 0	(other)
Mx 	30.3506
My 	77.1061
T	0 0 0 0	(name)
Mx 	50.3506
T	0	(.)
My 	89.1061
T	0 0 0 0	(SILE)
Mx 	77.1162
T	0 0 0 0 0	(comes)
Mx 	108.8818
T	0 0 0 0	(with)
Mx 	135.6474
T	0	(a)
Mx 	147.4131
T	0 0 0 0 0 0	(number)
Mx 	184.1787
T	0 0	(of)
Mx 	200.9443
T	0 0 0 0 0 0 0 0	(packages)
Mx 	247.7100
T	0 0 0 0 0	(which)
Mx 	279.4756
T	0 0 0 0 0 0 0	(provide)
Mx 	30.3506
My 	101.1061
T	0 0 0 0 0 0 0 0 0 0	(additional)
Mx 	88.5381
T	0 0 0 0 0 0 0 0 0 0 0 0 0	(functionality)
Mx 	153.5381
T	0	(.)
Mx 	166.7256
T	0 0 0 0 0	(Lorem)
Mx 	199.9131
T	0 0 0 0 0	(ipsum)
Mx 	233.1006
T	0 0 0 0 0	(dolor)
Mx 	266.2881
T	0 0 0	(sit)
Mx 	289.4756
T	0 0 0 0	(amet)
Mx 	309.4756
T	0	(,)
Mx 	30.3506
My 	113.1061
T	0 0 0 0 0 0 0 0 0 0 0	(consectetur)
Mx 	91.6541
T	0 0 0 0 0 0 0 0 0 0 0	(adipisicing)
Mx 	152.9577
T	0 0 0 0	(elit)
Mx 	172.9577
T	0	(,)
Mx 	184.2613
T	0 0 0	(sed)
Mx 	205.5649
T	0 0	(do)
Mx 	221.8684
T	0 0 0 0 0 0 0	(eiusmod)
Mx 	263.1720
T	0 0 0 0 0 0	(tempor)
Mx 	299.4756
T	0 0	(in)
Mx 	309.4756
T	0	(-)
Mx 	30.3506
My 	125.1061
T	0 0	(ci)
Mx 	40.3506
T	0 0 0 0 0 0	(didunt)
Mx 	77.4756
T	0 0	(ut)
Mx 	94.6006
T	0 0 0 0 0 0	(labore)
Mx 	131.7256
T	0 0	(et)
Mx 	148.8506
T	0 0 0 0 0 0	(dolore)
Mx 	185.9756
T	0 0 0 0 0	(magna)
Mx 	218.1006
T	0 0 0 0 0 0	(aliqua)
Mx 	248.1006
T	0	(.)
Mx 	260.2256
T	0 0	(Ut)
Mx 	277.3506
T	0 0 0 0	(enim)
Mx 	304.4756
T	0 0	(ad)
Mx 	30.3506
My 	137.1061
T	0 0 0 0 0	(minim)
Mx 	61.8714
T	0 0 0 0 0 0	(veniam)
Mx 	91.8714
T	0	(,)
Mx 	103.3922
T	0 0 0 0	(quis)
Mx 	129.9131
T	0 0 0 0 0 0 0	(nostrud)
Mx 	171.4339
T	0 0 0 0 0 0 0 0 0 0 0 0	(exercitation)
Mx 	237.9548
T	0 0 0 0 0 0 0	(ullamco)
Mx 	279.4756
T	0 0 0 0 0 0 0	(laboris)
Mx 	30.3506
My 	149.1061
T	0 0 0 0	(nisi)
Mx 	55.8089
T	0 0	(ut)
Mx 	71.2672
T	0 0 0 0 0 0 0	(aliquip)
Mx 	111.7256
T	0 0	(ex)
Mx 	127.1839
T	0 0	(ea)
Mx 	142.6422
T	0 0 0 0 0 0 0	(commodo)
Mx 	183.1006
T	0 0 0 0 0 0 0 0 0	(consequat)
Mx 	228.1006
T	0	(.)
Mx 	238.5589
T	0 0 0 0	(Duis)
Mx 	264.0173
T	0 0 0 0	(aute)
Mx 	289.4756
T	0 0 0 0 0	(irure)
Mx 	30.3506
My 	161.1061
T	0 0 0 0 0	(dolor)
Mx 	63.0827
T	0 0	(in)
Mx 	80.8148
T	0 0 0 0 0 0 0 0 0 0 0 0 0	(reprehenderit)
Mx 	153.5470
T	0 0	(in)
Mx 	171.2791
T	0 0 0 0 0 0 0 0 0	(voluptate)
Mx 	224.0113
T	0 0 0 0 0	(velit)
Mx 	256.7434
T	0 0 0 0	(esse)
Mx 	284.4756
T	0 0 0 0 0 0	(cillum)
Mx 	30.3506
My 	173.1061
T	0 0 0 0 0 0	(dolore)
Mx 	65.9398
T	0 0	(eu)
Mx 	81.5291
T	0 0 0 0 0 0	(fugiat)
Mx 	117.1184
T	0 0 0 0 0	(nulla)
Mx 	147.7077
T	0 0 0 0 0 0 0 0	(pariatur)
Mx 	187.7077
T	0	(.)
Mx 	198.2970
T	0 0 0 0 0 0 0 0 0	(Excepteur)
Mx 	248.8863
T	0 0 0 0	(sint)
Mx 	274.4756
T	0 0 0 0 0 0 0 0	(occaecat)
Mx 	30.3506
My 	185.1061
T	0 0 0 0 0 0 0 0 0	(cupidatat)
Mx 	82.7412
T	0 0 0	(non)
Mx 	105.1318
T	0 0 0 0 0 0 0 0	(proident)
Mx 	145.1318
T	0	(,)
Mx 	157.5224
T	0 0 0 0	(sunt)
Mx 	184.9131
T	0 0	(in)
Mx 	202.3037
T	0 0 0 0 0	(culpa)
Mx 	234.6943
T	0 0 0	(qui)
Mx 	257.0850
T	0 0 0 0 0 0 0	(officia)
Mx 	299.4756
T	0 0	(de)
Mx 	309.4756
T	0	(-)
Mx 	30.3506
My 	197.1061
T	0 0 0 0 0 0	(serunt)
Mx 	66.3778
T	0 0 0 0 0 0	(mollit)
Mx 	102.4050
T	0 0 0 0	(anim)
Mx 	128.4323
T	0 0	(id)
Mx 	144.4595
T	0 0 0	(est)
Mx 	165.4868
T	0 0 0 0 0 0 0	(laborum)
Mx 	200.4868
T	0	(.)
Mx 	30.3506
My 	209.1061
T	0 0 0 0	(SILE)
Mx 	55.3506
T	0 0 0 0 0	(comes)
Mx 	85.3506
T	0 0 0 0	(with)
Mx 	110.3506
T	0	(a)
Mx 	120.3506
T	0 0 0 0 0 0	(number)
Mx 	155.3506
T	0 0	(of)
Mx 	170.3506
T	0 0 0 0 0 0 0 0	(packages)
Mx 	215.3506
T	0 0 0 0 0	(which)
Mx 	245.3506
T	0 0 0 0 0 0 0	(provide)
Mx 	30.3506
My 	221.1061
T	0 0 0 0 0 0 0 0 0 0	(additional)
Mx 	85.3506
T	0 0 0 0 0 0 0 0 0 0 0 0 0	(functionality)
Mx 	150.3506
T	0	(.)
Mx 	160.3506
T	0 0 0 0 0	(Lorem)
Mx 	190.3506
T	0 0 0 0 0	(ipsum)
Mx 	220.3506
T	0 0 0 0 0	(dolor)
Mx 	250.3506
T	0 0 0	(sit)
Mx 	270.3506
T	0 0 0 0	(amet)
Mx 	290.3506
T	0	(,)
Mx 	30.3506
My 	233.1061
T	0 0 0 0 0 0 0 0 0 0 0	(consectetur)
Mx 	90.3506
T	0 0 0 0 0 0 0 0 0 0 0	(adipisicing)
Mx 	150.3506
T	0 0 0 0	(elit)
Mx 	170.3506
T	0	(,)
Mx 	180.3506
T	0 0 0	(sed)
Mx 	200.3506
T	0 0	(do)
Mx 	215.3506
T	0 0 0 0 0 0 0	(eiusmod)
Mx 	255.3506
T	0 0 0 0 0 0	(tempor)
Mx 	30.3506
My 	245.1061
T	0 0 0 0 0 0 0 0 0 0	(incididunt)
Mx 	85.3506
T	0 0	(ut)
Mx 	100.3506
T	0 0 0 0 0 0	(labore)
Mx 	135.3506
T	0 0	(et)
Mx 	150.3506
T	0 0 0 0 0 0	(dolore)
Mx 	185.3506
T	0 0 0 0 0	(magna)
Mx 	215.3506
T	0 0 0 0 0 0	(aliqua)
Mx 	245.3506
T	0	(.)
Mx 	255.3506
T	0 0	(Ut)
Mx 	270.3506
T	0 0 0 0	(enim)
Mx 	295.3506
T	0 0	(ad)
Mx 	30.3506
My 	257.1061
T	0 0 0 0 0	(minim)
Mx 	60.3506
T	0 0 0 0 0 0	(veniam)
Mx 	90.3506
T	0	(,)
Mx 	100.3506
T	0 0 0 0	(quis)
Mx 	125.3506
T	0 0 0 0 0 0 0	(nostrud)
Mx 	165.3506
T	0 0 0 0 0 0 0 0 0 0 0 0	(exercitation)
Mx 	230.3506
T	0 0 0 0 0 0 0	(ullamco)
Mx 	270.3506
T	0 0 0 0 0 0 0	(laboris)
Mx 	30.3506
My 	269.1061
T	0 0 0 0	(nisi)
Mx 	55.3506
T	0 0	(ut)
Mx 	70.3506
T	0 0 0 0 0 0 0	(aliquip)
Mx 	110.3506
T	0 0	(ex)
Mx 	125.3506
T	0 0	(ea)
Mx 	140.3506
T	0 0 0 0 0 0 0	(commodo)
Mx 	180.3506
T	0 0 0 0 0 0 0 0 0	(consequat)
Mx 	225.3506
T	0	(.)
Mx 	235.3506
T	0 0 0 0	(Duis)
Mx 	260.3506
T	0 0 0 0	(aute)
Mx 	285.3506
T	0 0 0 0 0	(irure)
Mx 	30.3506
My 	281.1061
T	0 0 0 0 0	(dolor)
Mx 	60.3506
T	0 0	(in)
Mx 	75.3506
T	0 0 0 0 0 0 0 0 0 0 0 0 0	(reprehenderit)
Mx 	145.3506
T	0 0	(in)
Mx 	160.3506
T	0 0 0 0 0 0 0 0 0	(voluptate)
Mx 	210.3506
T	0 0 0 0 0	(velit)
Mx 	240.3506
T	0 0 0 0	(esse)
Mx 	265.3506
T	0 0 0 0 0 0	(cillum)
Mx 	30.3506
My 	293.1061
T	0 0 0 0 0 0	(dolore)
Mx 	65.3506
T	0 0	(eu)
Mx 	80.3506
T	0 0 0 0 0 0	(fugiat)
Mx 	115.3506
T	0 0 0 0 0	(nulla)
Mx 	145.3506
T	0 0 0 0 0 0 0 0	(pariatur)
Mx 	185.3506
T	0	(.)
Mx 	195.3506
T	0 0 0 0 0 0 0 0 0	(Excepteur)
Mx 	245.3506
T	0 0 0 0	(sint)
Mx 	270.3506
T	0 0 0 0 0 0 0 0	(occaecat)
Mx 	30.3506
My 	305.1061
T	0 0 0 0 0 0 0 0 0	(cupidatat)
Mx 	80.3506
T	0 0 0	(non)
Mx 	100.3506
T	0 0 0 0 0 0 0 0	(proident)
Mx 	140.3506
T	0	(,)
Mx 	150.3506
T	0 0 0 0	(sunt)
Mx 	175.3506
T	0 0	(in)
Mx 	190.3506
T	0 0 0 0 0	(culpa)
Mx 	220.3506
T	0 0 0	(qui)
Mx 	240.3506
T	0 0 0 0 0 0 0	(officia)
Mx 	30.3506
My 	317.1061
T	0 0 0 0 0 0 0 0	(deserunt)
Mx 	75.3506
T	0 0 0 0 0 0	(mollit)
Mx 	110.3506
T	0 0 0 0	(anim)
Mx 	135.3506
T	0 0	(id)
Mx 	150.3506
T	0 0 0	(est)
Mx 	170.3506
T	0 0 0 0 0 0 0	(laborum)
Mx 	205.3506
T	0	(.)
[1] Mx 	169.9131
My 	484.3672
T	0	(1)
End page
Finish

Harfbuzz version: 0.9.40
Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
C:/Windows/Fonts/GentiumPlus-R.ttf:0	Unknown version

The fonts should be good as I can use them with the old version from the same install without issues (I just switch the DLLs and checkout older SILE commit), so it looks like Harfbuzz issue as the results are wrong in my viewer as well (no PDF output).

[simoncozens]

Yeah, all the zeroes are telling me that it's a Harfbuzz issue. But the fact that Harfbuzz can't find any glyphs and the opentype parser can't read the version number makes me suspect something else: harfbuzzShaper:getFace isn't actually picking up the right data - possibly not any data at all. Could you try this on the SILE REPL:

> #(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)

You should get about 362644. If you don't, have a look at:

> SILE.shaper:getFace(SILE.font.loadDefaults({}))

The code I'm suspecting, which looks very innocuous, is:

local fh,e = io.open(face.filename)
if e then SU.error("Can't open "..e) end
face.data = fh:read("*all")

[pkulchenko]

#(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)
You should get about 362644. If you don't, have a look at:

I'm getting 91:

> #(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)
91
> SILE.shaper:getFace(SILE.font.loadDefaults({}))
{data= ☺   ↕☺  ♦  LTSH°↓ç╤  ☺,  ♠ºOS/2qócU  ╘   VVDMXgWn┌,  ♣αcmap\♫☺╫  ♫♀  ♦Çcvt ☺åG  ↕î   ,family=Gentium,filename=C:/Windows/Fonts/Gentium-R.ttf,fullname=Gentium,index=0,pointsize=12}
> SILE.font.loadDefaults({})
{direction=LTR,family=Gentium Plus,features=,hyphenchar=-,language=en,script=,size=10,style=,variant=normal,weight=400}

This looks like I only have Gentium installed and nothing else? I checked fonts.conf and the cache that was generated and it seems to report all the fonts that I have installed. Is there anything else I should check?

[pkulchenko]

hm, maybe it needs to read as binary file...

[pkulchenko]

Yep, this fixes it:

diff --git a/core/harfbuzz-shaper.lua b/core/harfbuzz-shaper.lua
index 7a5710a..504705f 100644
--- a/core/harfbuzz-shaper.lua
+++ b/core/harfbuzz-shaper.lua
@@ -53,7 +53,7 @@ SILE.shapers.harfbuzz = SILE.shapers.base {
     local face = fontconfig._face(opts)
     SU.debug("fonts", "Resolved font family "..opts.family.." -> "..(face and face.filename))
     if not face.filename then SU.error("Couldn't find face "..opts.family) end
-    local fh,e = io.open(face.filename)
+    local fh,e = io.open(face.filename, "rb")
     if e then SU.error("Can't open "..e) end
     face.data = fh:read("*all")
     return face

[pkulchenko]

Simon, everything is back to normal so far with the fix, but I noticed one small difference between rules shown in the generated PDF and my preview:

This is how the line information is shown in the output:

Draw line	126.51168904979	160.39748101271	28.346457	0.708661425
Draw line	154.85814604979	160.39748101271	28.346457	0.708661425

The lines are generated using the following: \hrule[width=10mm,height=0.25mm].

Not sure why the line is moved relative to the text, but I've been wondering which one is the correct location. Note that it's quite possible there is something wrong with my positions, but all the other objects (including the image) are properly positioned on the page.

[alerque]

The generated PDF is correct, your preview is off. The height parameter is supposed to specify the thickness of the line, not the raise distance relative to the baseline. Something is off there.

[pkulchenko]

Interesting; thank you Caleb. I'll check my calculations...

[pkulchenko]

@alerque, thank you for checking. It turned out that the rule calculations are correct, but I used to have a small adjustment for the text when I was trying to match the PDF output and it was only done for text. Essentially, when I remove that adjustment, everything looks the same as in the PDF, but it's somehow 2 pixels higher than it should be. Is there a way to check for exact pixel coordinates in generated PDF to match with the ones I'm calculating?

Maybe there is something wrong with the point relative to which my text is drawn? I calculate it as the following:

  local function l2d(v) return v*16/12 end
--
  setFont = function (options)
      local w, h, descent, leading = mdc:GetTextExtent("Text", font)
      fontadj = h-descent
-- 
  moveTo = function (x,y)
    cx = x
    cy = y
  end,
-- and then
  outputHbox = function (value,w)
      mdc:DrawText(value.text, l2d(cx), l2d(cy)-fontadj)

Because the text drawing in my case is relative to the top left corner, I recalculate it from the baseline origin by subtracting height-descent. The rule is in the baseline now as expected, but somehow the entire text is about 2 pixels higher than it should be (comparing to the images that are positioned in absolute coordinates on the page). Any idea why this happens?

[harrysummer]

I think I have done pretty much of the tasks in the main content of this issue. Checkout my fork here: https://github.com/harrysummer/sile/tree/win32

• The reproducible Windows build: this is handled by using CMake and Azure Pipeline for continuous integration.
• The entry script is wrapped into exe by using srlua -- a small piece of code that concats an binary with lua source code. The binary would look for the lua script in itself.
• Isolate the libraries and ship distribution: all dependencies are built in the CMake script and installed along with SILE. The CI build artifacts can be downloaded here

[harrysummer]

Just want to share with you the workspace on Windows I am using. It is quite fascinating that we can configure to use TeXstudio for editting SILE, and then previewing side-by-side.

[akavel]

@harrysummer Oh wow, that is a brilliant idea!

[simoncozens]

Wow. This is superb.

[goyalyashpal]

I don't think the build artifacts alone are going to get us something that someone can install and use
@simoncozens at comment

so. what have we got now? have we got a portlable zip which i can extract and run it? i'd be okay with it even if it doen't install and even if i may have to configure some folder locations thereafter somewhere if that's what'd be needed.

The Resolved one

• We just made this harder by requiring ICU
  @simoncozens at comment

what's ICU? International Componenets for Unicode? https://icu.unicode.org
edit: yeah, verified from the link : ICU libraries in readme

[alerque]

so. what have we got now? have we got a portlable zip which i can extract and run it?

We had something like that, but at this point I don't know if the things inside the ZIP are usabel.

It is being built in CI by Azure and the builds are still running (successfully) for most commits and PRs to the project, but none of us are in a position to evaluate them and figure out what works, what doesn't, what needs changing, etc.

You can see recent builds here: https://dev.azure.com/sile-typesetter/sile/_build

It claims to have built an artifact, but what is or isn't in that artifact at this point is anybody's guess.

[KrasnayaPloshchad]

The idea behind a C wrapper (which I've now written) ties into my end goal for distribution on Windows: rather than installing Cygwin and Lua and Luarocks and everything else and then SILE, the installer just unpacks a punch of stuff into C:\Program Files\SILE, where all the dependency libraries - Mingw, Freetype, Harfbuzz, ICU, Lua and so on - are stashed away and nobody needs to care that they're running Lua and half of Unix when they type "sile foo.sil".

This could be easily done via NSIS.

[alerque]

I have no idea what NSIS is, but sure knock yourself out. I'm happy to facilitate contributions from Windows folks making their life easier, I just don't have the interest or time to make it happen myself.

Also medium term there is a very solid chance one option for running SILE will be as a single Rust binary with an embeded Lua interpreter and all the default support files baked in. See #1762 for current progress on that front. Getting the Rust CLI and embeded Lua interpreter is already fully working, but getting everything including the assorted C modules embeded is a bit trickier.

[KrasnayaPloshchad]

I have no idea what NSIS is, but sure knock yourself out. I'm happy to facilitate contributions from Windows folks making their life easier, I just don't have the interest or time to make it happen myself.

That’s shocking. Right now, you can acknowledge NSIS here, which give great help for this issue: https://nsis.sourceforge.io/Main_Page

[Omikhleia]

Why would that be shocking?

AFAIC, I've used NSIS since the early 2000s for some job-related projects.
Packaging is only a small part of the problem, though -- last time I could use a Windows machine (more than one year ago), the artifacts from the above-mentioned Azure CI build were lacking things, and I couldn't make them work. I didn't investigate a lot, but it all boils down to what @alerque already said:

It claims to have built an artifact, but what is or isn't in that artifact at this point is anybody's guess.

Someone interested and thinking it's an easy topic is surely welcome to help 😸