Paragraph indentation is lost after commands

[pkulchenko]

This can be seen on this document:

\begin[class=book,papersize=129mm x 198mm]{document}
\script[src=packages/rules]
I have seldom heard him mention her under any other name.

\hrule[width=100mm,height=0.25mm]

More text here
\end{document}

More text here continues on the same line as the horizontal rule, but should start on a new line.

I've bisected it to this commit 269963a, more specifically to lineEndLineStartSpace in command =((P("\\")-P("\\begin")) * Cg(myID, "tag") * Cg(parameters,"attr") * V"bracketed_stuff"^0 * lineEndLineStartSpace)-P("\\end{").

[simoncozens]

Urgh. Working out which spaces should and shouldn't be ignored after commands seems like it is beyond me. :-(

A\relax␠B

should render as A B. But

A\relax
B

should render as AB, and (presumably) so should

A\relax
B

and possibly even

A\relax␠ (additional space after \relax)
B

But

A\relax
␠
B

should render as

A

B

(I think.) If this is correct, and someone could put the rule into words, we can write tests for it. At the moment I'm just too confused. :-(

[pkulchenko]

I think the logic should be absolutely simple: \command\s*EOL should be removed. This means that for the purpose of formatting, if the line ends with command and some optional spaces, the command is removed (along with the spaces and the end-of-line) as if it never existed in the text.

I think it should handle nicely all the test cases you have in your example, but I tried to change lineEndLineStartSpace to (lpeg.S(" ")^0 * lpeg.S("\r\n"))^-1, which expresses what I'm looking for, but it still doesn't work. Any idea why?

[pkulchenko]

I think I got it; this is the diff against the current master:

diff --git a/core/inputs-texlike.lua b/core/inputs-texlike.lua
index 4805208..fe7eb11 100644
--- a/core/inputs-texlike.lua
+++ b/core/inputs-texlike.lua
@@ -14,7 +14,7 @@ SILE.inputs.TeXlike.parser = function (_ENV)
   local list = lpeg.Cf(lpeg.Ct("") * pair^0, rawset)
   local parameters = (P("[") * list * P("]")) ^-1 / function (a) return type(a)=="table" and a or {} end
   local anything = C( (1-lpeg.S("\\{}%\r\n")) ^1)
-  local lineEndLineStartSpace = (lpeg.S(" ")^0 * lpeg.S("\r\n")^1 * lpeg.S(" ")^0)^-1
+  local lineEndLineStartSpace = (lpeg.S(" ")^0 * lpeg.S("\r\n"))^-1

   START "document";
   document = V("stuff") * (-1 + E("Unexpected character at end of input"))
@@ -22,10 +22,10 @@ SILE.inputs.TeXlike.parser = function (_ENV)
   stuff = Cg(V"environment" +
     ((P("%") * (1-lpeg.S("\r\n"))^0 * lpeg.S("\r\n")^-1) /function () return "" end) -- Don't bother telling me about comments
     + V("text") + V"bracketed_stuff" + V"command")^0
-  bracketed_stuff = P"{" * V"stuff" * (P"}" + E("} expected"))
-  command =((P("\\")-P("\\begin")) * Cg(myID, "tag") * Cg(parameters,"attr") * V"bracketed_stuff"^0 * lineEndLineStartSpace)-P("\\end{")
+  bracketed_stuff = P"{" * V"stuff" * (P"}" + E("} expected")) * lineEndLineStartSpace
+  command =((P("\\")-P("\\begin")) * Cg(myID, "tag") * Cg(parameters,"attr") * V"bracketed_stuff"^0)-P("\\end{")
   environment =
-    P("\\begin") * Cg(parameters, "attr") * P("{") * Cg(myID, "tag") * P("}") * lineEndLineStartSpace
+    P("\\begin") * Cg(parameters, "attr") * P("{") * Cg(myID, "tag") * P("}")
       * V("stuff")
     * (P("\\end{") * (
       Cmt(myID * Cb("tag"), function(s,i,a,b) return a==b end) +

[alerque]

It isn't just paragraph breaks that are a problem here. There is a problem with trailing commands on all lines:

two
words

…typesets as "two words" whether or not you include a trailing space on the first line. On the other hand:

\em{two}
words

…typesets crammed together as "_two_words", again whether or not you manually include any spaces after the command.

See one of the Greek reader page samples from @jtauber for an example of this being a problem.

[pkulchenko]

Good point. @simoncozens, is it possible to identify those commands that produce any output in the document? Maybe, as a post-processing step, drop empty lines that had commands on them?

The logic would have only 3 steps:

1. Drop any trailing whitespaces after a command (but not the new line)
2. Capture the fact that the line has a command
3. Delete any empty line if it had a command

It can be simplified further by removing not just empty lines, but also whitespace only lines that include commands, but I'm not sure about all implications. One good aspect is that it would make _\command, \command, and \command_ work the same, but I'm not sure if there are any cases where the difference may be significant.

[pkulchenko]

@alerque, I'm still thinking about this and it may be the case that \em{two}\nwords behavior is fine. It's difficult for me to say, but the patch that I shown earlier handles the cases that Simon listed exactly as he wanted to.

There are several easy workarounds: (1) add a space before words, (2) put those on one line, or (3) add a comment after \em: \em{two} %. Each of those should render two and words separately.

I'd prefer to handle this case "automatically", but it may not be possible without deeper analysis.

@simoncozens, thoughts?

[simoncozens]

Right now my thoughts are I would like to rip out the changes I made and return to the old, you-need-to-add-percent-signs-everywhere parser. It was less user-friendly but at least it was consistent and unsurprising...

[alerque]

😢 Having more %s floating around than even LaTeX needed was not a happy state of affairs.

[pkulchenko]

It was less user-friendly but at least it was consistent and unsurprising...

I wouldn't call it unsurprising as I was quickly bitten by text\n\command\nmore text not being in the same paragraph (and I'm quite familiar with LaTeX). I have a habit of putting commands (especially with a list of parameters on a separate line), which breaks paragraphs with the current logic.

Having more %s floating around than even LaTeX needed was not a happy state of affairs.

I'm with Caleb on this one. This is one of the things I'd prefer to see less in my text/code.

@simoncozens, I'm proposing a very simple rule that should (1) handle the examples you listed, (2) handle those examples that Caleb mentioned, and (3) be simple to implement:

When a line starts with a command without any leading spaces, treat the previous new line as a space unless it's an empty line (a line without non-whitespace characters). That's it.

This means that text \command and text\n\command will be rendered the same, which is what the user would want in most (all?) cases. The "unless it's an empty line" clause is needed to avoid merging the line into the first paragraph in the following:

Some text

\hrule[width=100mm,height=0.25mm]

More text

[pkulchenko]

I've spent half a day today trying to figure out what exactly the problem is and why the current solution is not working.

After trying the various examples and fixes, I came full circle. I agree with @simoncozens that the earlier functionality needs to be restored (revert 269963a and 069205f).

If the functionality is restored, there are several "features" that (ideally) needs to be fixed:

• several commands before a paragraph change indentation of that paragraph. This is the issue described in Bigger paragraph indent when using \par instead of new line #54.
• several commands with an empty line produce a standalone (empty) paragraph.
• several commands at the top of the document with an empty line produce a standalone (empty) paragraph (this and the previous item are the same issue, just caused by different scenarios).

Examples:

\begin[class=book,papersize=129mm x 198mm]{document}
\script[src=packages/color]
\script[src=packages/rules]%\hrule[width=10mm,height=0.25mm]
\script[src=packages/image]

First word, first paragraph.

\raggedright
\raggedright
Last paragraph
\end{document}

I've tried various fixes, but I don't think any of them are going to work simply because there are commands that need to ignore spaces and other commands that don't need to and the parser doesn't have that information. For example, in the example above \hrule may need to be rendered and if there are two \hrule commands, then the whitespaces between them become significant.

Even the rendering of that \hrule is not without flaws as the previous command includes a whitespace, so the rule is rendered with a small indent. Adding % to the second line (with \script[src=packages/color]) fixes the issue.

If it's not possible to come up with a rule that is going to work automatically, I think we may need to introduce a new command that will mark a set of commands and "eat" all whitespaces. This is still better than % as it's going to be applied to all inserted commands and everything else in between.

I'm proposing \cmd and \cmd{}. The former will ignore all whitespaces from this command to the end of the paragraph or the first non-empty text (whichever comes first). The latter will ignore whitespaces inside its body.

I've tried to implement this command based on font command, but I couldn't figure out how to walk content to remove/ignore whitespaces inside:

    SILE.registerCommand("cmd", function(options, content)
      if (type(content)=="function" or content[1]) then
        SILE.settings.pushState()
        SILE.process(content)
        SILE.settings.popState()
      end
    end, "Processes commands while ignoring whitespaces")

Any help?

[alerque]

there are commands that need to ignore spaces and other commands that don't need to and the parser doesn't have that information

I've been playing with scenarios myself and arrived at much the same conclusion you did. The parser needs to be fed more information. It's a crazy hard problem, but however it is solved I want a syntax I can actually teach to non-programmers and have them understand what the deal is. Trailing comment operators to solve a white space parsing issue seems like it will be a never ending debacle to explain.

The \cmd{} (or perhaps a more descriptive name like\silent{} or \nooutput{}) makes a bit of sense and I agree is an improvement over trailing %. That would also allow for block syntax to be used, ala <head> it HTML or everything before \begin{document} in LaTeX without worrying about throwing off the document layout with some extra spaces before the content.

\begin{cmdonly}
    \stuff
    \here
\end{cmdonly} 

(Still trying on command names for size there.)

Another possibility that occurred to be is simply an alternate syntax for all commands. If the idea is that the parser doesn't have enough information, some structural part of the syntax could act as a flag for whether or not any given command should or should not (and perhaps how) eat whitespace.

[pkulchenko]

The \cmd{} (or perhaps a more descriptive name like\silent{} or \nooutput{}) makes a bit of sense and I agree is an improvement over trailing %. That would also allow for block syntax to be used, ala it HTML or everything before \begin{document} in LaTeX without worrying about throwing off the document layout with some extra spaces before the content.

Right; I also played with \quiet, \preamble and few others; I prefer \cmd because (1) it's similar to \script and other SILE tags (it's a container for commands) and (2) it's short.

I didn't like \quiet, \silent, or \nooutput, because this command doesn't suppress output from commands; it only removes whitespaces between commands.

\trim is another option as it trims the result ("to make (something) neat by cutting it"), but \cmd may be more similar to the philosophy behind SILE tags.

Another possibility that occurred to be is simply an alternate syntax for all commands. If the idea is that the parser doesn't have enough information, some structural part of the syntax could act as a flag for whether or not any given command should or should not (and perhaps how) eat whitespace.

I prefer a specific tag, simply because it limits the number of commands and because it eats whitespaces between commands, which would still be difficult to explain by having alternative syntax.

I got it working as a container \cmd{}, but I also want to get it working as \cmd (to the beginning of the text), similar to how \font works and can't figure out how to do it properly yet.

[pkulchenko]

@simoncozens, @alerque, I've implemented \cmd command in both \cmd and \cmd{} variants; pushed to https://github.com/pkulchenko/sile/tree/command-friendly-whitespace. This branch also includes a revert of the two WIP commits in SILE.

Simon, I simply hacked massage_ast as I couldn't make it work with command handlers for \cmd (as it needs to be applied for subsequent elements that are not included in content passed to the command handler). I'm sure you have some mechanism for traversing the tree, but I couldn't find it, so for ended up using the AST for testing purposes.

It achieves all the effects I wanted to; the following example works as I expect it:

\begin[class=book,papersize=129mm x 198mm]{document}\cmd
\script[src=packages/color]
\script[src=packages/rules]%\hrule[width=10mm,height=0.25mm]
\script[src=packages/image]

First word, first paragraph.

\cmd{
\raggedright
\raggedright}

Last paragraph
\end{document}

The first \cmd (on the first line) eliminates incorrect vertical spacing and indentation for the first paragraph. The second \cmd{} eliminates vertical spacing between paragraphs.

If you want to see the "original/current" behavior, simply comment out line 97 in inputs-texlike.lua.

I played with making it enabled by default, but don't think it's a good idea.

[pkulchenko]

Simon, what do you think about the \cmd proposal?

[pkulchenko]

@simoncozens, @alerque, any further thoughts on this? The current state is sort of the middle of not-quite-tex-like and not-quite-working-in-some-cases. The \cmd proposal would revert this to the original behavior (which is expected result for those familiar with TeX), but provide a way to achieve proper formatting around commands without % galore.

[simoncozens]

Hi Paul, sorry about the delay; I've been prioritizing a lot of the internationalisation work and stuff with discretionaries - now that's sorted and the tests pass again I'd like to come back to this. I think the \cmd proposal is the way forward - revert the misguided changes to the parser and then introduce a new process function which doesn't pass stuff to the typesetter at top level.

[pkulchenko]

This sounds good. You can check my attempt to do this (https://github.com/pkulchenko/sile/tree/command-friendly-whitespace) as it includes the revert of the changes (one commit), plus the new command (another commit); I'm sure you can find a proper way to implement the needed logic ;). I've tried to make it work with both \cmd and \cmd{} and would like to have both variants available; the former is a shortcut that works for one paragraph (as far as I remember).

[simoncozens]

I've come up with another way to do this which I think works OK. Basically we revert the two erroneous changes and introduce a new rule: text which consists entirely of newlines is dropped. Please have a look at the latest commit, especially the new test tests/space-after-command.sil. If this works the way everyone expects, I will fix up the regression test expectations.

[alerque]

Hmmmm, that's very interesting. So far all the random tests I through at it did the thing I expected.

[alerque]

Seeing as how you've jinxed me twice today I'm asking: @simoncozens are you working on the tests or would you like me to fix them up? I have a previous test related commit to send and can send it with or without updated regression tests for this issue.

[simoncozens]

I wasn't going to fix up the tests for this yet - was looking for a third opinion from @pkulchenko before declaring this done.

[alerque]

Alright well I have a fixed up set (all but one that I'm not sure about) that I'll post in a minute but you can wait to merge it until we decide on this. If it turn out we do something different I'll split out the non-whitespace related changes from the set.

[alerque]

One case where this handling is broken (relative to previous behavior) is in the \pagetemplate command. Indenting such as:

\pagetemplate[…]{
    \frame[…]
}

…leaves blank space at the top of a document where it didn't used to while

\pagetemplate[…]{
\frame[…]
}

…works as expected.

[simoncozens]

I toyed with the idea of never typesetting anything which is purely whitespace, but then you have the (very useful) \book:section:post and similar macros which are defined as being a space. \book:sectioning\book:section:post{}Section name should insert a space. So that wouldn't work.

I think the way around this is to give \pagetemplate and other purely structural commands their own version of SILE.process that never sends any text to the typesetter.

[raphCode]

I noticed these two don't produce the same output:

\font[size=15]{
aaa
}

bbb

text appears in two lines

\begin[size=15]{font}
aaa
\end{font}

bbb

text appears in one line

In any case, I expected two lines.
I am not sure if this a "new" test case or if it can be reduced to one of the previous scenarios reported here.

Would the approach of "discarding whitespace around non-output commands" yield the correct result here? Is \end{font} considered an outputting command?

[Omikhleia]

@raphCode Yep... Weird parsing logic that we all face sooner or later (my very first issue here was quite similar, #1193)... Not to mention internal spaces, &c. Any change would be quite breaking, though, for lots of packages likely, and I am a bit frightened this is now marked for "0.15" (with the usual milestone shifting though...). "Non-output commands" is a concept I find highly dubious to define strictly...

[alerque]

My milestone handling for "some future major version before 1.x" (as I see this issue fitting) vs. "scheduled for the next major version" has been inconsistent with how I've been marking and shifting minor version bumps to mean either "some future minor version" vs. "scheduled for the next minor version". I'll try to rectify that for consistency so that "scheduled for exact version N" means what it says and "some future version level X" can get shifted around.

[alerque]

@raphCode The case of \font is probably one of the most egregious examples and the reason this issue is still open and not just marked as "won't fix". The problem is that \font was initially implemented with two different modes: when passed content it just applies a font to that content, when passed no content (empty) it changes the current typesetter font for all future content. The issue is that for consistency with other parsing (like the way we handled \script and others) the latter use case shouldn't automatically end a paragraph while the former may. We currently assume one use case without actually checking or knowing which is weird and arguably wrong.

[alerque]

It occurred to me today while walking to the train that this issue is basically defining whether our input grammar will be context-free or not. It's not precisely the same stakes but in the ballpark. If the command itself determines whether whitespace will be eaten or not we loose a large element of parsability / predictability even if in some cases it would be more expected at first blush.

We have four syntax variants to consider (\foo, \foo[…], \foo{…}, \begin{foo}…\end{foo}) with potential to handle them differently (and more subtly form 3 and 4 could be with or without content visible at the parser level). But if it matters if foo() is a noop or a utility function or a typesetting function that outputs something then we have a totally different kind of language. If whitespace handling is different for \noop, \script, \font, and \strong we have issues. I suggest to stay closer to the context-free end of things we must treat whitespace around commands the same for all commands whether they output anything or not. We may handle syntax forms 1 and 2 differently from 3 and 4 but the decision should be 100% determined by syntax not determined by the substance of the command.

[pkulchenko]

We may handle syntax forms 1 and 2 differently from 3 and 4 but the decision should be 100% determined by syntax not determined by the substance of the command.

I absolutely agree with this goal.

[alerque]

@raphCode

Quick question based on a variant of your comment.

Would you expect these two blurbs to work the same or different?

\font{aaa}

bbb

Vs.

\begin{font}
aaa
\end{font}

bbb

[raphCode]

Assuming that an empty line translates into a new paragraph, I would expect them to work the same, or am I missing something?
I am not sure if the \font without options does something useful, but does it matter for the question?

But really, as long as there is any system that is consistent and easy to use for the majority of use cases, I think I am fine.

[raphCode]

Another thing I noticed is that the following behaves differently in TeX and Sile:

\begin{document}
hello \relax \relax \relax world
\end{document}

Sile creates multiple spaces between the words, TeX doesn't.

While this is a artificial example, in a real document something more subtle can happen:

text here
\todo{todo notes like in LaTeX}
more text here

Before printing, the \todo is usually declared as a no-op. With the current state of Sile this creates double spaces inside paragraphs.

---

Edit:
Nevermind, TeX gets this wrong too, meaning double spaces in the middle of my paragraphs where todo notes are inserted, even when the package gets disabled.

[raphCode]

I found these links interesting, they discuss whitespace handling after commands in TeX:
https://tex.stackexchange.com/questions/31091/space-after-latex-commands
https://tex.stackexchange.com/questions/25820/no-space-following-macro-without-argument
https://tex.stackexchange.com/questions/86565/drawbacks-of-xspace

---

I think we should pick some space-swallowing defaults for the 4 different forms of commands (as outlined by alerque) and additionally offer a way in the code to make this behavior overrideable. With careful consideration which built-in commands to override, this should offer a tradeoff between predictability and comfort of use. Ideally, it just works the way one expects.

For example:

• \lorem and similar commands are expected to produce output, so they should not swallow spaces by default.
  \relax could be an exception since it does nothing. It can then make use of the override and discard whitespaces afterwards. This also fits the test cases in this thread better.
• \em{content} and similar are likely to produce typesetting content, so by default these commands may also not swallow spaces.
  A todo notes packages could override this default to enable space swallowing, so the double space problem in my previous post can be avoided. This is reasonable since todo notes only add content to the margin, not the main content frame where this command appears.
• \relax and other \commands could probably swallow spaces by default.
  Except if I define a Sile logo with \sile, I want it to behave more like a normal word in typesetting, not eating spaces afterwards. This is where an override seems very handy. It would also elegantly avoid all the pitfalls of xspace or weird command definitions to work around TeX space handling like the stackexchange links above.

I realize that this proposal is probably not straightforward to implement since the question how to deal with a certain whitespace can not be answered purely based on parser grammar.

[alerque]

• \relax and other \commands could probably swallow spaces by default.

Out of curiosity, what are you using \relax for in your SILE projects and (without looking at the sources) can you summarize what you think it does?

[raphCode]

what are you using \relax for

I don't - this was just an example I picked up by reading this thread.

I already looked the \relax command up as soon as I read it here, in generally it does nothing. For TeX it seems to have usages related to command parsing / delimiting iirc. In SILE it is implemented as an empty function and I wondered why it is even there, hoping that it is not needed by having a better/different parser than TeX.

I figured that the details of \relax do not matter in this discussion, so I contributed without having a real clue what it does. Maybe I am wrong and I am missing important aspects of some commands.
If you feel that my posts here do not add value to the discussion, please tell me so I can back off or educate myself :)

Edit: I replaced my first bullet point with a better example, and kept the original one strike-trough to not disturb your quote.

[Omikhleia]

Kind of related (from #1691)

\begin{pullquote}
Some quote
\end{pullquote}

leads to an extra white space at the start of the quoted text. (It may perhaps hint, or not, towards ignoring trailing spaces at the end of an environment start)

EDIT "or not": depends on how we consider the parity with XML.

<pullquote>aaa</pullquote>

Maybe not being the same thing as

<pullquote>
aaa
</pullquote>

[alerque]

Even though my comments there start heading the other way, I'm not 100% convinced the issue in #1691 is the same. In this case the difference between

<foo>bar</foo>

and

<foo>
bar
</foo>

should never equate to

<foo> bar</foo>

But only whether the outcome is

SILE.process({ "bar" })

or possibly

SILE.typesetter:leaveHmode()
SILE.process({ "bar" })

Does that make sense?

[Omikhleia]

I'm not 100% convinced the issue in #1691 is the same

Indeed, it's not the same as "paragraph" issues initially discussed here. To clarify, my mentioning here as "kind of related" is that whatever we adopt in the future for the paragraphing issue (incl. blank lines / end of lines), it might be good also to consider how it possibly affects spacing. But at the core, it's possibly a bit distinct, true.

(I am not sure, by the way, that the outcome should depend on whether horizontal mode is left or not. It's a tough question, though.)

[alerque]

This one has really started to bother me, even sitting up at 4am with a 10mo who can't breathe I'm stewing on how to fix it.

I started work on a grammar. That's helping to clarify what the issues is, but the input grammar isn't the full story.