A framework for building lexers. See examples/json for a working example on how to build a JSON lexer with this framework.
Go get it with
go get -u github.com/TimSatke/lexer
Define your own states as you can see in examples/json/lexer.go.
- You want to build a parser and need a lexer, and don't want to build the base yourself, because it is effort to build and test it, and this framework already did that part
Eventually, yes. At the moment though, no. This is due to my lack of experience with parsers and their architecture and design. Feel free to create one using this framework, though.
First, you create a lexer. Second, you pass a starting state. This state will be executed (a state is a function that can drain/accept runes from the lexer's input), and must return the next state to go into.
The base lexer implementation provided with this package works on a byte slice.
It has two markers, start and pos. pos is the current position in the
input byte slice, start is the position, where the last token was emitted.
Upon emit, start will be set to pos, marking the start position of the next
token.
Whenever the lexer encounters unexpected runes, it is recommended to emit an
error token and nil as the next state. This will stop the lexer to process the
input. See
this
as an example on how to emit errors. The error tokens can then be processed by
your parser, which will consume the lexer's token stream (retrieve it with
lexer.TokenStream().Tokens()).