UTF-8 ?

[drzraf]

From: algorithms/french/stem_ISO_8859_1.sbl

stringdef a^   hex 'E2'  // a-circumflex
stringdef a`   hex 'E0'  // a-grave
stringdef c,   hex 'E7'  // c-cedilla

stringdef e"   hex 'EB'  // e-diaeresis (rare)
stringdef e'   hex 'E9'  // e-acute
stringdef e^   hex 'EA'  // e-circumflex
stringdef e`   hex 'E8'  // e-grave
stringdef i"   hex 'EF'  // i-diaeresis
stringdef i^   hex 'EE'  // i-circumflex
stringdef o^   hex 'F4'  // o-circumflex
stringdef u^   hex 'FB'  // u-circumflex
stringdef u`   hex 'F9'  // u-grave

So far there is no UTF-8 version. Why?

[ojwb]

There is, but it's byte-for-byte identical (because ISO-8859-1 encodings exactly match the first 256 codepoints from Unicode) so rather than have two copies in git to keep in sync the build system handles it.

[drzraf]

Would you wanto to make these files publicly available so that other projects (snowball-js => lurn-languages [JSON=>UTF-8]) could take them as a reference?

[ojwb]

Would you wanto to make these files publicly available so that other projects (snowball-js => lurn-languages [JSON=>UTF-8]) could take them as a reference?

All the files involved are already in the git repo.

[drzraf]

I meant, the generated, UTF-8, corresponding files. Eg: algorithms/french only contains ISO_8859_1 and MS_DOS_Latin_I files but not the "generated" UTF-8 counterparts. Don't you think it would make sense?

[ojwb]

As I said above, the Unicode version of the Snowball source is byte-for-byte identical to the ISO_8859_1 version when there's an ISO_8859_1 version.

If that's really too complicated, you can just get the build system to create all the Unicode versions like so:

for d in algorithms/* ; do make "$d"/stem_Unicode.sbl ; done

[drzraf]

How is "é" (C3A9) considered then given stringdef e' hex 'E9' // e-acute
If it's not, then how is it going to work for an UTF-8 string?

[ojwb]

It's handled as Unicode codepoint U+00E9 when the snowball compiler is running in "widechars" or UTF-8 mode (which are enabled automatically for some backends - e.g. only "widechars" makes sense for Java, and only UTF-8 for Go).

[drzraf]

Got it. Sorry for the above misleading "c3a9" UTF-8 and thank you ojwb for the clarifications!
This one is right then as is this one
But I ended up using this one (carry.js)