Syntax Extensibility

[hildjj]

I've been talking about syntax extensibility, most recently on #239 and #291 (comment). I'd like to come up with a relatively scalable way for us to add new features in such a way that we don't need a new ASCII character to mark each feature. I would also like to ensure that we have room to parameterize rules (see: pegjs/pegjs#36 and pegjs/pegjs#45) and any other templating features we want eventually.

I really like the way that Pug does mixins. They allow you to pass in formal parameters, other blocks of Pug code, and generic name/value pairs that can be used inside the mixin.

I wonder if we could figure out a syntax that would be similarly useful, but fit within the existing Peggy grammar. In particular, we need at least postfix (like ?) and prefix (like !) rules... we could make the expression before and the expression after available, therefore allowing prefix, postfix, or infix as needed. You might need some extra parameters as well.

If there was a good enough syntax for this, we could use it for built-ins as well. Perhaps built-ins would start with a capital letter or something to reserve a namespace?

[reverofevil]

we need at least postfix (like ?) and prefix (like !) rules

First I'd like to ask, do we? The most impressive PEG DSL I've seen was Parsec, especially if used in Applicative style. The only place it uses infix operators is where it has to overload languages' function application (that isn't an issue for a custom language such as Peggy), and the rest is implemented as regular function calls.

There are a few reasons prefix and postfix notation in Peggy is beneficial:

• compatibility with regular expressions syntax, which beyond any doubt provides better DX than opt, some and some1;
• brevity where it's most needed, such as prefix $ or @.

I don't think there's a lot of missing things that should look like operators. (Well, {n,m} would be nice, but the topic of "how many projects actually used PEG.js with CoffeeScript, and should we care" is better discussed elsewhere.)

Skepticism aside, there is such a concept as mixfix notation. Generating grammars for it is discussed in linked article. (And it's a good read anyway. Have you ever heard that except "prefix", "postfix" and "infix" there's also a forth option of "closed"?)

If there was a good enough syntax for this,

In my experiments, there was significant whitespace. a (b) and a(b) are considered different expressions: sequence and parametrized parser correspondingly. Never caused any problems for me, but I'm unsure about DX for other users. (I think it worked this way here).

Given there are already places in Peggy's grammar where whitespace is significant ([a-z]i vs [a-z] i), probably it's not an issue at all.

Perhaps built-ins would start

Built-ins are not really needed when there is a way to import. It's just the same thing, but one of the libraries is provided by Peggy.

[Mingun]

Generally speaking, I doubt that this problem is so extensive that it requires separate discussion. In fact, there are only 4 features that we may want to add to the grammar:

• repetition syntax
• import syntax
• templates syntax
• attributes syntax

In principle, the last 3 features could be solved by attributes syntax, but I think, it would be better to have dedicated syntax for each of them.

I doubt that something else may be needed that cannot be expressed by the attribute.