TASK: Implement alternative approach to lexical analysis

[grebaldi]

solves: #3, #33

This PR introduces a Lexer class that implements a very different approach to lexical analysis than the existent Tokenizer:

• it uses generalized matchers
  • (most) matchers are stateless and can tell whether a given character at a given offset matches their specification
    • only exception is Sequence that uses state to switch between multiple subsequent matchers
• it works on-demand
  • parsers specify which token types they expect next, the lexer then differentiates only within that given set
• it handles multi-byte characters
  • multi-byte characters are interpreted correctly, and counted as one character each
  • this is achieved without the use of expensive mb_* functions

I focused somewhat on memory-efficiency and expect that the implementation will be more economical on memory use than the Tokenizer.

Token types have also changed to a more rigid set. This will simplify some of the parser implementations later on.

As of right now, I'm not too sure of this approach and expect things to break when I turn to the parser implementations. I'm also not quite sure if I haven't missed anything on the multi-byte character handling (seems too easy to me 😅). It'll require more tests further down the line to be on the safe side with this. For now, all of this is just an experiment.

Remaining TODOs

• Implement new Lexer
• Reform all parsers to use new Lexer
• Remove Tokenizer and all other old concepts
• Refactor Lexer and increase test coverage
• Parse negative integers both in expression and integer literal
• TemplateLiteral lines must have indentation >= of that of their TemplateLiteralNode

[mhsdesign]

Wait a second, isnt this similar to the philosophy of the new fusion parser? ❤️

edit it seems more sophisticated ^^ would love to talk about this ^^

[mhsdesign]

interestring. because we allow the space token here its not a TEMPLATE_LITERAL_CONTENT (but it probably should be in real world ^^)

[grebaldi]

Yep, the TemplateLiteralParser will read this differently :)

[mhsdesign]

hmm im just wondering what are the pros and cons of making this thing mutable ...

on the one side, the lexer can expose it as public readonly member but methods like override and reset might always be smelly. Then again, this mutable buffer might be a performance optimization, as we dont need a new object every time.