Proposal: default token types

[nathan]

Upon further consideration, #88 seems like a good idea. Markdown(ish) syntaxes are the obvious motivating example: arbitrary text with certain embedded sequences have special meanings, and incomplete special sequences should be passed through verbatim.

const lexer = moo.compile({
  para: {lineBreaks: true, match: /(?:\r?\n|\r){2,}/},
  issu: {match: /#\d+/, value: s => s.slice(1)},
  lstr: /\*\*(?=\S)|__(?=\S)/,
  rstr: /\*\*(?=\s|$)|__(?=\s|$)/,
  escp: {match: /\\./, value: s => s.slice(1)},
  text: moo.default,
})

lexer.reset(`
Upon **further consideration,** #88 seems like a good idea.

Markdown(ish) syntaxes are the obvious motivating example…
`.trim())

console.log([...lexer]) /*
[ { type: 'text', value: 'Upon ' },
  { type: 'lstr', value: '**' },
  { type: 'text', value: 'further consideration,' },
  { type: 'rstr', value: '**' },
  { type: 'text', value: ' ' },
  { type: 'issu', value: '88' },
  { type: 'text', value: ' seems like a good idea.' },
  { type: 'para', value: '\n\n' },
  { type: 'text', value: 'Markdown(ish) syntaxes are the obvious motivating example…' } ]
*/

@tjvr Feel free to bikeshed the name. (fill might be better?)

[moranje]

Sounds great to me, this will make the parsing of my text based language a lot easier. Two things:

1. The name moo.default might clash with when the node module is being read as an ES6 module e.g. const moo = require('moo').default. I would go with defaultToken but anything is okay really.
2. Is the order important when specifying a default token?

[nathan]

The name moo.default might clash with when the node module is being read as an ES6 module

AFAIK the de facto rule is to do that conservatively or not at all when the exports don't contain __esModule: true. But again, there's probably a better name than default regardless.

Is the order important when specifying a default token?

No; this matches the behavior of moo.error.

[tjvr]

This is a great idea! I'll need to think about the name. 😊

[tjvr]

This is great; thank you for writing it!

I'm concerned it makes things a bit more complicated, but I think that's okay; I'd just like to review it carefully.

Could we try renaming default to fallback? I'll then have a good look at this! 🙌

[tjvr]

Why do we have both shouldThrow and error?

[nathan]

error is like fallback but it consumes until the end of the buffer instead of just to the next valid token. shouldThrow throws a syntax error instead of returning the error token.

[tjvr]

I note this will conflict with your other PR 😞

[nathan]

Rebased.

[tjvr]

minor: I think I'd prefer queuedToken

[nathan]

Renamed.

[tjvr]

Just to confirm: will /foo|bar/g try and match foo at each index in the buffer, and only once that fails, attempt to match bar? (Which would be bad.)

_{Sent with GitHawk}

[nathan]

@tjvr No. The exec algorithm explicitly works by attempting to match the RegExp at each string index (AdvanceStringIndex is just +1 for non-unicode RegExps), so it will find the earliest match of any complete path through the RegExp, including the current lastIndex if there is a match there.

[tjvr]

I love how simple this is. ❤️

[moranje]

Just tested this on my codebase, it's working as intended.