Add listeners on any token

[piuccio]

Instead of providing only a callback for code highlight, open the possibility to call a custom function for any token type.

This can be done with the configuration option on.

For instance

on : {
   heading : function () {}
}

I'm using this to generate anchors next to headings and the table of contents automatically.

var toc = [];
marked(input, {
    on: {
        heading : function (token, callback) {
            if (token.depth <= 3) {
                toc.push({
                    depth: token.depth,
                    title: token.text
                });
                callback(null, "<a name='" + token.text.toLowerCase().replace(/[ ]+/g, "_") + "'></a>" + token.text);
            } else {
                callback(null, token.text);
            }
        }
    }
}, function (err, content) {
    // do something with my content and TOC
});

[leeola]

So, i'm working on making a branch of this code work with Wintersmith, but it appears your code is missing one thing. The ability for non-callback based code working with marked(), correct?

That is to say, your code will only run if a callback is given to marked().

Correct me if i am wrong, i am still quite new to this codebase. As of yet, i'm unsure why the callback and non-callback sections seem to drastically different heh.

Thoughts?

[piuccio]

Indeed
This pull request is more a proof of concept rather than real code, I don't expect it to be integrated as it is.

Now it only works for the callback version and when highlight is enabled.

I'm very new to this code base and I needed a quick solution, but if the idea and the approach are good I can try to spend more time and implement it correctly, including unit tests.

It would be nice to hear from @chjj as he's working as well on extensibility

[scottgonzalez]

I suppose the goal here is to avoid walking the list of tokens one extra time? You'll still need to parse the TOC separately and merge the results. I'm interested to see if anything gets added here, because we do TOC generation for jQuery sites as well.

[chjj]

This raises a good point: there's no way to alter the tokens if you're using marked async. I think I would rather move the current async behavior into the Parser instead of the marked() function. This way you can use Lexer.lex to get the tokens, alter them however you want, and then call Parser.parse asynchronously. Or, just have marked() accept tokens as input when it is async.

[ChrisWren]

I like this idea. I submitted a PR #247 a few days ago adding custom header markup support, but this is more extensible as it applies to all token types. I am willing to help out to make this PR work as I really want custom header support to get GitHub hover links on headers. I am going to spend some time today looking at the source in reference to @chjj's comment.

[ChrisWren]

As I understand from the source code, the highlight option is the only async option. It seems that async highlight is used after Lexer.lex to perform additional token processing before Parser.parse is called on the tokens. If async highlighting isn't enabled, these lines of code are executed to do synchronous highlighting in the Parser.

@chjj is the idea that we would keep all the preprocessing synchronous and then see if the user is using the asynchronous version of marked when calling Parser.parse synchronously or asynchronously?

[joshbruce]

Closing as stale and no tests.