feat: flatten childToken arrays

[UziTech]

Marked version: 11.1.1

Description

Flatten custom extension childToken arrays before running them through walk tokens. This will allow custom extensions to have child tokens like the table token where the rows field is an array of arrays of tokens.

• Fixes Register Walktokens Behavior for extensions with different token structure #3170

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
marked-website	✅ Ready (Inspect)	Visit Preview	Jan 20, 2024 8:02pm

[UziTech]

@calculuschild this should work to walk arrays of arrays of tokens.

One question: Should we flatten with infinite depth? or should we only flatten 1 or 2 levels deep?

The only issue I found with doing Infinite depth is there can be a stack overflow if there is a circular dependency. I don't think that is a big issue but something to think about.

a = [1]
b = [2]
a.push(b)
b.push(a)
a.flat(Infinity) 

[calculuschild]

I think infinite depth shouldn't be a problem. If people are designing extensions in such a way to cause a cyclic dependency, that's a fault of the extension, not Marked, right?

[UziTech]

I agree. This shouldn't affect core marked at all. Only extensions

[styfle]

I think you could also use a large-ish depth instead of Infinity, perhaps 1000, and still cover most use cases.

But I agree, its only for self-referencing extensions which is probably a small minority.