What's required for a extension that handles cross-line content?

[07akioni]

I'm building an extension that may have cross-line content, for example replace any {...} pattern into __demo__:

1. {foo} => __demo__ (this is easy)
2. {\nfoo\n} => __demo__ (this is not easy, since there are line breaks)

After I made an extension, I got this error in second case:

      throw new RangeError('Cannot access index `' + index + '` in a splice buffer of size `' + (this.left.length + this.right.length) + '`');
            ^

RangeError: Cannot access index `8` in a splice buffer of size `8`
    at SpliceBuffer.get (file:///Users/zlc/dev/micromark-extension-demo/node_modules/.pnpm/micromark-util-subtokenize@2.0.1/node_modules/micromark-util-subtokenize/lib/splice-buffer.js:55:13)
    at subcontent (file:///Users/zlc/dev/micromark-extension-demo/node_modules/.pnpm/micromark-util-subtokenize@2.0.1/node_modules/micromark-util-subtokenize/index.js:145:19)
    at subtokenize (file:///Users/zlc/dev/micromark-extension-demo/node_modules/.pnpm/micromark-util-subtokenize@2.0.1/node_modules/micromark-util-subtokenize/index.js:71:30)
    at postprocess (file:///Users/zlc/dev/micromark-extension-demo/node_modules/.pnpm/micromark@4.0.0/node_modules/micromark/lib/postprocess.js:12:11)
    at micromark (file:///Users/zlc/dev/micromark-extension-demo/node_modules/.pnpm/micromark@4.0.0/node_modules/micromark/index.js:58:5)
    at file:///Users/zlc/dev/micromark-extension-demo/src/index.mjs:50:13
    at ModuleJob.run (node:internal/modules/esm/module_job:217:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:308:24)
    at async loadESM (node:internal/process/esm_loader:42:7)
    at async handleMainPromise (node:internal/modules/run_main:66:12)

The extension's core logic is:

const demoConstruct = { name: "demo", tokenize: demoTokenize };

// 123: {
// 125: }

export const demos = { text: { 123: demoConstruct } };

function demoTokenize(effects, ok, nok) {
  return start;

  function start(code) {
    effects.enter("demo");
    effects.consume(code);
    return inside;
  }

  function inside(code) {
    if (code === 125) {
      effects.consume(code);
      effects.exit("demo");
      return ok;
    }
    if (code === null) {
      return nok;
    }
    effects.consume(code);
    return inside;
  }
}

I wonder what's happened in this case? I want to handle line break correctly, what should I do?

I've read some other extensions (for example extension math) but can't find what is the crucial point.

Hopefully someone can help me. Thanks.

[wooorm]

Hi!

First off, I don‘t understand your replace value, why would anyone want __demo__ to appear in a document?

What you are making looks a bit like the example from the readme. I hope you saw that. When you see that, you might also see that it’s included in further exercises. And, earlier, that it’s mentioned that micromark-extension-mdx-expression is similar but more advanced, already: https://github.com/micromark/micromark-extension-mdx-expression.

Looking at its source code, you will see that line endings are in a separate token: https://github.com/micromark/micromark-extension-mdx-expression/blob/9f7875c314ea0b4603331aa46595af83594f9163/packages/micromark-factory-mdx-expression/dev/index.js#L142-L147.
This is the main thing you are missing:

1. you need a wrapper token (which you can call demo), it contains other tokens, you do not consume directly in it
2. then you can stuff regular characters into a demoData or so
3. and line endings go into a lineEnding.

This is important for markdown, because other things can exist here. Take for example:

* > asd {foo
  > bar} qwe

That indentation, those block quote markers, have to go somewhere. That can only work if regular characters and line endings are in separate tokens.

[07akioni]

uld anyone want __demo__

This is just a simplification for my scene. Actually I want to replace \(equation\) or \[equation\] into a latex equation.

[07akioni]

Many thanks, I'll dig into it.