Unsupported textmate grammar (that contains non-advancing match rules)

[darkxeno]

This textmate grammar:
https://github.com/Benvie/JavaScriptNext.tmLanguage/blob/master/JavaScriptNext.tmLanguage

Doesn't work, shows the following error:
(command line)
[3400:0401/162730:INFO:CONSOLE(1270)] "Grammar is in an endless loop - case 3", source: (...)/vscode/node_modules/vscode-textmate/release/main.js (1270)

(chrome console, stacktrace)

Grammar is in an endless loop - case 3scanNext 

@ (...)/vscode/node_modules/vscode-textmate/release/main.js:127
0 (...)/vscode/node_modules/vscode-textmate/release/main.js:1270 Grammar is in an endless loop - case 3scanNext 
@ (...)/vscode/node_modules/vscode-textmate/release/main.js:1270_tokenizeString 
@ (...)/vscode/node_modules/vscode-textmate/release/main.js:1207Grammar.tokenizeLine 
@ (...)/vscode/node_modules/vscode-textmate/release/main.js:1067Tokenizer.tokenize 
@ TMSyntax.ts:223tokenize 
@ TMSyntax.ts:125_tokenizeLine 
@ textToHtmlTokenizer.ts:100_tokenizeLines 
@ textToHtmlTokenizer.ts:90_tokenizeToString 
@ textToHtmlTokenizer.ts:73tokenizeToString 
@ textToHtmlTokenizer.ts:17container.appendChild.htmlContentRenderer_1.renderHtml.codeBlockRenderer 
@ modesContentHover.ts:271_renderHtml 
@ htmlContentRenderer.ts:65renderHtml 
@ htmlContentRenderer.ts:35(anonymous function) 
@ modesContentHover.ts:247(anonymous function) 
@ modesContentHover.ts:246ModesContentHoverWidget._renderMessages 
@ modesContentHover.ts:226ModesContentHoverWidget._withResult 
@ modesContentHover.ts:212(anonymous function) 
@ modesContentHover.ts:141HoverOperation._onComplete 
@ hoverOperation.ts:143HoverOperation._triggerSyncComputation 
@ hoverOperation.ts:117(anonymous function) 
@ hoverOperation.ts:71RunOnceScheduler.onTimeout 
@ async.ts:566

[alexdima]

@darkxeno That message indicates a problem with the grammar.

A TM parser is basically an interpreter for the grammar on the passed in text.

Sometimes, grammars contain logical flaws.

The one auto-detected in this case is that a grammar contains some rules which leads to the case where the grammar is not advancing the text to be tokenized and also not transitioning to a different state.

This all depends on the text being tokenized, so if you wanna share the text that you try to tokenize I can try to find what specific rules in that specific grammar gets it to enter an endless loop.

[darkxeno]

For me happens on any JS code file when using that grammar. I send you an
example file: https://jsfiddle.net/8961kyu3/

This same grammar i being used in some sublime extensions, and as fas as i
know is seems to be working fine there.

I really love the colors that this grammar can obtain, would be wonderful
if you can make this to work.

Very thanks

2016-04-04 12:35 GMT+02:00 Alexandru Dima notifications@github.com:

@darkxeno https://github.com/darkxeno That message indicates a problem
with the grammar.

A TM parser is basically an interpreter for the grammar on the passed in
text.

Sometimes, grammars contain logical flaws.

The one detected in this case is that a grammar contains some rules which
leads to the case where the grammar is not advancing and also not
transitioning to a different state.

This all depends on the text being tokenized, so if you wanna share the
text that you try to tokenize I can try to find what specific rules in that
specific grammar gets it to enter an endless loop.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#9 (comment)

[darkxeno]

Any new about this issue?

Thanks

[bottleofwater]

I copy here my message from the first-mate repository (atom/first-mate#77 (comment)):

I've been able to track down the issue to this regexp (and others similar):

(?x)
  !(?!=)| # logical-not     right-to-left   right
  \|\| |  # logical-or      left-to-right   both

As you can see, these regexp have a trailing pipe that make them match everything. I'm not sure what to do - on one hand, it's obviously a coding mistake to leave this trailing pipe, but on the other hand it seems to work fine with both SublimeText and TextMate (I guess they discard every matching rule with 0-tokens as "not matching"), on a quite popular grammar. What do you think?

[bottleofwater]

The JavaScriptNext syntax apparently uses the official Sublime's PackageDev in order to generate its tmLanguage files from YAML grammars, and it seems that this tool is the one adding trailing pipes. Fixing this would require updating every tmLanguage file generated by this tool up to this point. It isn't really feasible, so I feel like that it should be fixed on the vscode-textmate codebase instead. My two cents.

My bad, the issue is actually in the YAML files themselves.

[alexdima]

@bottleofwater Thank you for the analysis. In this case, may I suggest to file an issue against the grammar itself. It should be trivial at that level to remove trailing |. We allow regular expressions to match 0 characters, many grammar rules use begin expressions that don't consume anything, we would need to "ignore" only match expressions that don't consume anything, which given node-oniguruma's scanner based API (used by both vscode-textmate and atom-firstmate) is not trivial.

[SDP190]

@darkxeno if you can tell me please how did you manage to add and use a TM grammar with Visual Studio Code other then the built-in?

[alexdima]

I have removed the console.error since this case is not an error