How to do discretionary/soft hyphens?

[jslabovitz]

I'm typesetting a book, and trying to fix the hyphenation in a paragraph. I've tried inserting soft-hyphen Unicode characters (U+00AD), as well as adding a custom hyphenation exception... but neither is working. Is it because the word I'm trying to hyphenation (Christchurch) a proper name? If so, is there any way to force this hyphenation? (That's what I tried with the soft-hyphen initially.) Or a setting to tweak?

EDIT: Sorry, I should have been more clear about the soft-hyphen: it actually does break the word, but doesn't add an actual hyphen character. See 2nd paragraph, below.

Here's my most recent try, with the exceptions (in XML):

<document class="book">

<language main="en"/>
<font size="14pt" language="en"/>

<script>
  SILE.typesetter:typeset(SILE.showHyphenationPoints("Christchurch", "en"))
</script>

<hyphenator:add-exceptions>Christ-church</hyphenator:add-exceptions>

<script>
  SILE.typesetter:typeset(SILE.showHyphenationPoints("Christchurch", "en"))
</script>

xxx xxx <!-- to hit hyphenation point -->
The calls and the fellowship are the same worldwide. Michael and I danced in England and Australia and New Zealand. Half a world away in Christchurch, New Zealand, a couple drove 45 minutes out of their way to collect us from our hotel and take us to a dance. We felt a true sense of belonging from the minute we walked in the door.

xxx xxx <!-- to hit hyphenation point -->
The calls and the fellowship are the same worldwide. Michael and I danced in England and Australia and New Zealand. Half a world away in Christ&#xAD;church, New Zealand, a couple drove 45 minutes out of their way to collect us from our hotel and take us to a dance. We felt a true sense of belonging from the minute we walked in the door.

</document>

[alerque]

The pattern system used by the hyphenator uses only lower-case patterns, but later uses a case-insensitive matcher to map to words from the input. Try this:

<hyphenator:add-exceptions>christ-church</hyphenator:add-exceptions>

[jslabovitz]

Aha! That worked. Thank you so much!

[jslabovitz]

Any idea why the soft-hyphen doesn't work as (I) expected, though?

[Omikhleia]

By the way, beyond the specific case of soft hyphens, as far as SIL or XML input are concerned, the same considerations could apply to other Unicode special characters, e.g. U+00A0 (non-breakable space).¹

Footnotes

1. You might perhaps find https://github.com/Omikhleia/markdown.sile/issues/46 interesting, to that respect. ↩

[Omikhleia]

On no-break space U+00A0 see PR #1918

On soft hyphen U+00AD I just looked into it after the above PR, and things are more complex than meet the eye...

First, most fonts have it at zero-width and no visual representation, but some fonts have a visual representation here or a width -- probably a bad decision, but it will then appear in the output in most places where it is used, which is not what one wants...

Regarding shaping:

• The character is only dropped at shaping if it occurs in a ligature (e.g. "dif-ficult" is shaped as "d i ffi c u l t", and the soft hyphen is lost)
• Otherwise it is passed through to the Unicode node maker, where ICU sees it, but we handle it as a penalty (due to the isBreaking condition)

Morevover, a word such as Christchurch may be broken by the ICU even without soft hyphen (with a soft break, resulting in a penalty). In that case, there won't be any hyphenation character '(= the word is broken, but without hyphen character)

I locally experimented adding it in the Unicode node maker (similarly to PR #1918):

• As noted, it's no longer here in ligatures, so it's only "partial"...
• It can be made to work otherwise, and processed as a discretionary, but:
  • Should it use the hyphenchar? This might be a language-dependent choice...
  • In language "und", it works well
  • In most other language with hyphenation patterns, the Liang hyphenation algorithm will run and insert additional discretionary nodes, so it's hard to say if the result would really be better than a neat hyphenation exception...

So to recap,

• The shaper loses some soft hyphens (= ignored)
• The ICU node maker replaces remaining soft hyphens by a penalty (= potential break, but no hyphen char)
• The ICU node maker, depending on language, may introduce soft line breaks (= potential break, but no hyphen char)
• The Liang hyphenation algorithm may add discretionary nodes (= secondary topic to keep in mind)

Feedback on these results is welcome (and I can push my experimental code on the node maker somewhere, maybe).

But with all these points, I am not even sure we would really want to support Unicode soft hyphens internally, at that low a level...
As an alternative,

• an inputter could, of course, decide to replace them with discretionary nodes, but that's not generic.
• it could be processed at typesetter level, even before shaping, though I am not very fond of that solution either. This might be the best workaround however...

[Omikhleia]

I can push my experimental code on the node maker somewhere, maybe

= Omikhleia@d69dbde

it could be processed at typesetter level, even before shaping, though I am not very fond of that solution either. This might be the best workaround however.

Well, SILE.typesetter:typeset() already does some pattern stuff (for typesetter.parseppattern), so it would add another special case here?

[Omikhleia]

Well, SILE.typesetter:typeset() already does some pattern stuff (for typesetter.parseppattern), so it would add another special case here?

For what is worth, this would be it:

diff --git a/typesetters/base.lua b/typesetters/base.lua
index 7b44b96c..6ac16885 100644
--- a/typesetters/base.lua
+++ b/typesetters/base.lua
@@ -276,7 +276,16 @@ function typesetter:typeset (text)
     if token.separator then
       self:endline()
     else
-      self:setpar(token.string)
+      for token2 in SU.gtoke(token.string, luautf8.char(0x00AD)) do
+        if token2.separator then -- soft hyphen support
+          local discretionary = SILE.nodefactory.discretionary({})
+          local hbox = SILE.typesetter:makeHbox({ SILE.settings:get("font.hyphenchar") })
+          discretionary.prebreak = { hbox }
+          table.insert(SILE.typesetter.state.nodes, discretionary)
+        else
+          self:setpar(token2.string)
+        end
+      end
     end
   end
   SILE.traceStack:pop(pId)

EDIT:

Pros:

• It does somehow work and is simpler that the Unicode node maker changes...

Cons:

• Ligatures won't be performed in the presence of a soft hyphen (no longer lost, but...)
• Some of the other concerns mentioned above still apply (choice of replacement, other discretionary added by the Liang algo)

[Omikhleia]

So for soft hyphens, see #1930 and related PR #1935 (with a few refinements over the earlier notes here)