Skip to content
This repository has been archived by the owner on Dec 15, 2022. It is now read-only.

Fix insertion of newline characters to the end of lines #100

Merged
merged 4 commits into from
Sep 22, 2017

Conversation

Ingramz
Copy link
Contributor

@Ingramz Ingramz commented Sep 21, 2017

This pull request fixes three things.

  1. Issue where recursive matches had a newline character appended to the subpatterns through Grammar.tokenizeLine (source for off-by-one error in Atom 1.18+ token matching behaving strangely atom#14982)
  2. Regression from version 7.0.3 where a branch in Rule.getNextTags was supplied with a line that contains newline character, although it expects one without (source for the other off-by-one error described in Showcase off-by-one error in language-html #99)
  3. Newline character was appended to the last line to be matched, although the last line never can contain one. This brings the behavior in line with TextMate (and possibly also Sublime Text).

If anyone asks, the newlines are appended so that it is possible to match them using patterns like $\n, as newlines are stripped when Grammar.tokenizeLines does splitting by \n. Currently by default Grammar.tokenizeLine is assumed to do the appending.

The branch in Rule.getNextTags expects a line without a newline because this is used to generate the tokens for the final output. If it contained a newline, then also the output tokens based off it would contain a newline. I would prefer if tokenizer output contained newlines, which would make possible to tokenize newlines (and for instance give a different right side scope at the end of the line), however this would be a breaking change as Atom does not expect newlines in tokenizer output and the API in general would change.

I believe this is the best fix we can have for now. I haven't tested this with Atom yet, because I couldn't figure out/have forgotten how to get these changes easily into a build.

Fixes #99
Fixes atom/atom#14982

@Ingramz
Copy link
Contributor Author

Ingramz commented Sep 21, 2017

Ping @50Wliu @maxbrunsfeld

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Atom 1.18+ token matching behaving strangely
3 participants