Adding a preprocessor to directives?

[mikeyhogarth]

I'm relatively new to the unified ecosystem (loving it) so this almost certainly has more to do with my lack of understanding, but I've scoured the docs and can't find an answer.

Basically, my users don't like the directive syntax so I'm trying to come up with a more subtle syntax for them to use. Something like this

A :nice[directive].
Some ((new:syntax)).   #  would become --> :new(value="syntax")

The idea was that I could transform the second line into directive syntax by just find/replacing the text, and then the normal remark-directives library would do the rest of the parsing. I got this far...

import remarkDirective from "remark-directive";
import remarkStringify from "remark-stringify";
import remarkParse from "remark-parse";
import { unified } from "unified";
import { findAndReplace } from "mdast-util-find-and-replace";

const content = `
A :nice[directive].
Some ((new:syntax)).
`;

const file = await unified()
  .use(remarkParse)
  .use(myRemarkPlugin)
  //.use(remarkDirective).  // <-- note that I have this commented out
  .use(remarkStringify)
  .process(content);

function myRemarkPlugin() {
  return function (tree) {
    findAndReplace(tree, [
      [
        /\(\(([a-zA-Z]+):([a-zA-Z]+)\)\)/g,
        function (full, prop, val) {
          return `:[${prop}]{ value="${val}"}`;
        },
      ],
    ]);
  };
}

console.log(String(file));

If i run that as-is, I get:

A :nice\[directive].
Some :\[new]{ value="syntax"}.

Exactly what I would expect, given that remarkStringify would be escaping the opening square brackets. If I uncomment the remarkDirective plugin though...

A :nice[directive].
Some ((new:syntax)).

The cause of this is that the colon in new:syntax is being interpreted as a directive. Colons are going to come up quite a lot in the work I'm dong (part of a domain specific language I'm using) - I think i could get around that issue, if only i could understand why the directives transform has already happened by the time my plugin runs and maybe how to stop it from happening.

The part that is confusing me overall is that the remarkDirective plugin is included (via use) AFTER myRemarkPlugin, so how come it's already been in and changed the tree by the time it's gotten to myRemarkPlugin?

I was expecting more like this behavior;

const file = await unified()
  .use(remarkParse)
  .use(myRemarkPlugin).  // changes the custom (( )) sytnax into a directive via text-replace
  .use(remarkDirective).   // thing from last step can now be interpreted as directive
  .use(remarkStringify)
  .process(content);

Thanks again for this awesome ecosystem/tooling and apologies if this is a stupid question / has been asked a million times. I did have a look and found a few people who looked like they were having similar problems but the threads all petered out.

Similar-ish issues

• Parsing of inline directive collides with usage of colon in natural language. (E.g. n:th) remark-directive#5
• this.use() doesn't respect plugin order when combined with unified().use() unifiedjs/unified#190

[ChristianMurphy]

Welcome @mikeyhogarth! 👋
There are two types of plugins, parser plugins and AST plugins.
parser plugins always run at the very beginning, remark-gfm and remark-directive are examples of this.
AST plugins run later and can work on the parsed content, your plugin is an example of one of these.

If you want to write a parser plugin, see

• Why you may not want to do this https://github.com/micromark/micromark#extending-markdown
• How to do this if you decide it is critical to do so https://github.com/micromark/micromark#creating-a-micromark-extension

[mikeyhogarth]

@ChristianMurphy you are an absolute legend, thank you!

These both point to micromark - i still don't think I understand the relationship between micromark and remark (I though that was an either/or type relationship)..

I'm guessng that micromark is the parser and it's going on under the hood of remark. I think this advice you're giving here is missing from the documentation - if I eventually figure this out, do you think there's an opening anywhere for some documentaiton contribution to help future explorers? Would be happy to write up the findings.

Specifically the part about there being 2 kinds of plugins , and generally interacting with the parser.

(or is this in the docs and I just missed it?)

[ChristianMurphy]

micromark is the tokenizer, tagging each character, remark converts that to an AST a tree structure.
remark builds on micromark, not everyone needs the AST abstraction, so some use micromark directly.

There are some references in the readme, I'm not sure how much more to add, customizing is generally not recommended.

[wooorm]

Specifically the part about there being 2 kinds of plugins

The different plugins is mentioned in several places; the main place is the Plugin type/docs in the unified repo.

and generally interacting with the parser.

So, this isn’t really recommended much, as Christian mentioned, which is why how to interact with it isn’t front and center.
Still, that micromark is used, is mentioned in the remark readme a couple of times: https://github.com/remarkjs/remark. Which also points to remark-parse, which includes even more info!