RBNF: Parser Generator for Context Sensitive Grammars

Tutorials

See Jupyter Notebook.

Preview

• rbnf.zero: solution to processing complex texts. Similar to re.

  See source file Poly.rbnf.

  # parse polynomials from text.
  import rbnf.zero as ze
  ze_exp = ze.compile("import poly.[*]", use='Poly')
  print(ze_exp.match("2x^2 + 3 + 4 - 7 x^5 + 4 x + 5 x ^2 - x + 7 x ^ 11").result)

  Then we got output [(0, 7), (1, 3), (2, 7), (5, -7), (11, 7)].

• CLI tool chains:

  λ rbnf.exe --help
  Available commands:
   cc
       rbnf source code compiler.
  
   - filename(positional or keyword arg)        : input source file
   - output(positional or keyword arg) = None   : output file name. default to be ...
   - name(positional or keyword arg) = 'unname' : name of language
  
   run
       You can apply immediate tests on your parser.
       P.S: use `--opt` option takes longer starting time.
  
   - filename(positional or keyword arg)    : python file generated by `rbnf` command, or rbnf sour file
   - opt(positional or keyword arg) = False : optimize switch
  
  
  
  λ printf "ignore [space]\nspace := R'\s+'\nterm := R'[^\s\(\)]+'\nsexpr ::= '(' sexpr* ')' | term" > lisp.rbnf
  λ rbnf run lisp
  
  type `:i` to switch between python mode and parsing mode.
  The last result of parsing is stored as symbol `res`.
  
  runbnf> (add 1 (add 2))
  
  sexpr[
        Tokenizer(name='auto_const', value='(', lineno=0, colno=0)
        sexpr[
              Tokenizer(name='term', value='add', lineno=0, colno=1)
        ]
        sexpr[
              Tokenizer(name='term', value='1', lineno=0, colno=5)
        ]
        sexpr[
               Tokenizer(name='auto_const', value='(', lineno=0, colno=7)
               sexpr[
                     Tokenizer(name='term', value='add', lineno=0, colno=8)
               ]
               sexpr[
                     Tokenizer(name='term', value='2', lineno=0, colno=12)
               ]
               Tokenizer(name='auto_const', value=')', lineno=0, colno=13)
        ]
        Tokenizer(name='auto_const', value=')', lineno=0, colno=14)
  ]
  
  runbnf> exit
  
  λ rbnf cc lisp.bnf -output lisp.py
  ...

Related

RBNF is leveraged to handle parsing tasks in some projects. Feel free to check them to get a better knowledge:

• Yet Another Python Python:

  A Python bytecode compiler to provide custom extended syntaxes and optimization passes.
  This provides a parser for extended Python(compatible to Python 3.6.x-3.8.0).

  {"a" : a, "b": {"c": c}} = {"a": 1, "b": {"c": 2, **other2}, **others1}
  assert a == 1 and c == 2

• Reley

  A haskell-like language implemented in Python. This provides a parser for a part of haskell.

    import operator (add, eq)
    import functools (reduce)
    import toolz (curry)
    import reley.prelude ((+))
  
    infix 5 (==)
    infix 0 ($)
    (==) = curry eq
    ($) a b = a b
    (+) = curry add
  
  
    m_sum lst = if lst == [] then 0
            else destruct lst
            where
                destruct (a, b) = a + m_sum(b)

• ReFining:

  A type checker.

  python cli.py
  
  reF> let x: 'a = 1 in
        fn x: 'a -> x ;;
  =>  (int`2 => int`2)
  

• RMalt:

  A concise demo of intepreter implementation.

  python test.py
  
  malt> let (%+) = (l, r) -> (l + r) % r;;
  malt> let f = (a, b) -> a  + b;;
  malt> f 1 2;;
  =>  3
  malt> f 1
        ;;
  =>  Fn(b)<id=3061211909288>
  malt> infix %+ 5;;
  =>  5
  malt> let (%+) = (l, r) -> (l % r) + r;;
  malt> 5 %+ 6;;
  =>  11
  
  

EDSL in Python

To be continue.

RBNF's compiler(rbnf -> python or others) relies on bootstrap.

Actually, in current version, bootstrap of RBNF is implemented in EDSL.

See rbnf bootstrap in Python.

A snippet could be previewed here.

rbnf = Language("RBNF")

...

@rbnf
class Primitive(Parser):

    @classmethod
    def bnf(cls):
        # @formatter:off
        return optimize(
                 C('(') + Or @ "or_" + C(')')
               | C('[') + Or @ "optional" + C(']')
               | Name @ "name"
               | Str  @ "str")
        # @formatter:on

     @classmethod
    def rewrite(cls, state: State):
        get = state.ctx.get
        or_: Parser = get('or_')
        optional: Parser = get('optional')
        ...

...

Other backend

To be continue.