Branching multiline syntax - modified micromark-extension-gfm-table

[TRIAEIOU]

Hello,
Thank all of you for the great work on micromark and all of unified!

Background

I am trying to modify the micromark-extension-gfm-table to also parse headerless tables (see end for samples). If possible I would like to avoid making "two copies of micromark-extension-gfm-table" modifying each a little to match a specific "table version". Rather, I would like to modify the micromark-extension-gfm-table so that it matches all the versions.

<not in table>
if row matches row: set state to temporary_row
else if row matches delimiter row: set state to delimiter_row
<next row>
if row matches delimiter row: change temporary_row to header_row,  set state to delimiter_row
else if row matches row: change temporary_row to body_row, set state to body_row
<next row>
parse body rows

Questions

1. Is attempt the correct way to go to first try one pattern, then another etc.? Or is there something else I should be looking at?
2. What is the correct use of attempt? I have understood the use of attempt as follows, however I can't seem to end up inside the "nok()" branch.

     {tokenize: tokenizerThatWillGoThroughTheWholeTableIfNoMismatchesDetected, partial: true},
     function (code: Code): void  | State {
          // Called if tokenizer called ok(), i.e. has consumed up to and including </tbody>
     },
     function (code: Code): void  | State {
          // Called if tokenizer called nok()
          // In my case I want to try "start parsing from delimiter row"
          // However I never seem to end up here
     });

Table variants

1. "Normal" GFM style table ⇒ <table><thead><tr><th>A</th><th>Normal</th></tr></thead><tbody><tr><td>GFM</td><td>Table</td></tr><tbody></table>
   |A |Normal|
   |-----|--------|
   |GFM|Table |

2. "Aligned headerless table" ⇒ <table><tbody><tr><td align="right">An</td><td>other</td></tr><tr><td align="right">Headerless</td><td>Table</td></tr><tbody></table>
   |-----------:|------|
   | An |other|
   |Headerless|Table|

3. "Headerless table" ⇒ <table><tbody><tr><td>An</td><td>other</td></tr><tr><td>Headerless</td><td>Table</td></tr><tbody></table>
   | An |other|
   |Headerless|Table|

[wooorm]

Thanks!

Rather, I would like to modify the micromark-extension-gfm-table so that it matches all the versions.

I am most likely against a PR to add support for non-GFM features to GFM extensions. It is already very hard to maintain extensions that match GFM. And tables are very complex. But GFM is necessary.
I also think extending markdown with new features is generally a bad idea. See the chapter on Extensions in the micromark readme for more of my thoughts.

// However I never seem to end up here

This is perhaps because actual tables are used first?

Is attempt the correct way to go to first try one pattern, then another etc.?

Attempt can indeed be used to try one path, and do something depending on whether that worked or not.

[TRIAEIOU]

• This is incorrect. The first that oks, oks the whole. If it noks, the next one will be attempted

Ok, perfect, that sounds like the way to go then, something like this then:

    return effects.attempt([
        {tokenize: tokenizeHeaderRow, partial: true}, // Try to tokenize as GFM table
        {tokenize: tokenizeDelimiterRow, partial: true}, // If GFM noks(), try to tokenize as delimiter_headerless
        {tokenize: tokenizeBody, partial: true}, // If delimiter_headerless noks(), try as body_only
      ],
      tableClose,
      function (code: Code): void | State {
        self.interrupt = originalInterrupt
        return nok(code)
      })(code);

Thanks!

[TRIAEIOU]

Hmm, it seems I am still not getting how attempt works.

With this as input:

|a|b|
|-|-|
|c|d|

Stepping through the following in the debugger with the following tokenizer:

function tokenizeTable(effects: Effects, ok: State, nok: State): State {
  // @ts-ignore
  const self = <Runtime>this;
  const align: Align[] = [];
  let tableColCount = 0
  let seenDelimiter: boolean | undefined;
  let hasDash: boolean | undefined;
  let inBody: boolean | undefined;
  return start;

  function start(code: Code): void | State {
    // @ts-ignore
    effects.enter('table')._align = align;

    return effects.attempt([
      {tokenize: tokenizeDelimiter, partial:true},
      {tokenize: tokenizeHeader, partial: true},
      {tokenize: tokenizeBody, partial: true}
      ],
      function (code: Code): void | State {
        return tableClose;
      },
      function (code: Code): void | State {
        //self.interrupt = originalInterrupt
        return nok(code)
      }
    )(code);
  }

When tokenizeDelimiter nok()s tokenizeHeader never gets called, the entire attempt exits and tokenizeTable gets called again on the next line. Does anyone have an idea on what I am missing?

Thanks

[wooorm]

You might have a bug in tokenizeDelimiter.
You should probably use the built in debugging logs, it helps: https://github.com/micromark/micromark#size--debug.

[TRIAEIOU]

To follow up (and maybe give some hints to anyone else wanting to make branching syntaxes), I got the chaining effects.attempt() and "stepping in at different levels of the table parsing" to work after realizing I needed to keep track of the object that the attempt was called on so that the correct instance was used when calling ok/nok (hard to explain but for anyone curious you can have a look at https://github.com/TRIAEIOU/micromark-extension-extended-table/blob/37668b5556f7b0cee8d023c688aa6daf38427952/dev/lib/syntax.js#L172 and onwards).

[TRIAEIOU]

To follow up again for anyone running into this issue - an example:

/**
 * @typedef {import('micromark-util-types').Extension} Extension
 * @typedef {import('micromark-util-types').Resolver} Resolver
 * @typedef {import('micromark-util-types').Tokenizer} Tokenizer
 */

/** @type {Extension} */
export const foo = {
  flow: {
    null: {
      tokenize: tokenizeFoo,
      resolve: resolveFoo
    }
  }
}

/** @type {Resolver} */
function resolveFoo(events, context) {
  return events
}

/** @type {Tokenizer} */
function tokenizeFoo(effects, ok, nok) {
  const _effects = effects
  const _ok = ok
  const _nok = nok
  return start

  function start(code) {
    return _effects.attempt(
      [
        {tokenize: tokenizeBar, partial: true},
        {tokenize: tokenizeBaz, partial: true},
      ],
      _ok,
      _nok
    )(code)
  }

  function tokenizeBar(effects_, ok_, nok_) {
    effects = effects_
    ok = ok_
    nok = nok_
    return startBar
  }

  function tokenizeBaz(effects_, ok_, nok_) {
    effects = effects_
    ok = ok_
    nok = nok_
    return startBaz
  }

  function startBar(code) {
    return nok(code)
  }

  function startBaz(code) {
    return ok(code)
  }
}

(I was updating some earlier code and had to spell it out for myself).