Scannerless parsing

[alllex]

In this approach to parsing, there is no separate tokenization process. Instead, each parser tries to directly parse a token it expects.

This has a number of advantages:

• The priority of tokens becomes local to the parser declaration (the order of components in the or sequence).
• Token declaration order becomes largerly irrelevant, which reduces the potential for user errors.
• Grammar composition and inheritance become more straightforward.

The main drawback for now is a slight drop in performance. This is due to inability to proactively extract the next token and make smarter work avoidance decisions (e.g. jumping directly to the right parser in the set of alternatives using the 1-token lookahead). However, this is a small price to pay for added flexibility and developer experience.

This is related and alleviates the problems described in

• Suggestion: Sort literal tokens by literal length #21
• Wildcard regex token after literal token does not work #22

[EndzeitBegins]

Hey @alllex, just wanted to let you know that I've tried the changes you made using a local build of parsus and from my point of view, it works like a charm now!

I can now easily declare my parser as a series of expected tokens or combinations thereof, basically just translating the specification document to Kotlin. I really, really like the declarativeness and composability this entails. Thank you for your work.

I'm looking forward for the next release.