v2

v2 was released in November 2020. It contains the following changes, some of which are backwards-incompatible:

Added optional LexString() and LexBytes() methods that lexer definitions can implement to fast-path lexing of bytes and strings.
A new stateful lexer has been added.
A filename must now be passed to all Parse*() and Lex*() methods.
The text/scanner lexer no longer automatically unquotes strings or supports arbitary length single quoted strings. The tokens it produces are identical to that of the text/scanner package. Use Unquote() to remove quotes.
Tok and EndTok will no longer be populated.
If a field named Token []lexer.Token exists it will be populated with the raw tokens that the node parsed from the lexer.

Support capturing directly into lexer.Token fields. eg.

type ast struct {
    Head lexer.Token   `@Ident`
    Tail []lexer.Token `@(Ident*)`
}

Add an experimental/codegen for stateful lexers. This provides ~10x performance improvement with zero garbage when lexing strings.
The regex lexer has been removed.
The ebnf lexer has been removed.
All future work on lexing will be put into the stateful lexer.
The need for DropToken has been removed.

Provide feedback