[typescript] use the language service for tokenization

[thorn0]

There is too many issues with the TM grammar, many of which seem to be impossible to fix.

E.g. in microsoft/TypeScript-TmLanguage#121, we can read this:

I think it's possible that this might never be completely fixed, due to limitations of the textmate grammar system.

Isn't it better to try another approach (the one that Visual Studio uses) instead of trying to fix the unfixable?

[aeschli]

I also think that tokenizers implemented by language servers is an interesting idea that we should evaluate again.
The current strategy is chosen for performance reasons. Tokenizing happens in the renderer process and is line base, optimized for colorizing as you type.
A language server tokenizer needs cross process communication, and likely analyses the whole file in order to not have the same problems as a text mate tokenizer: Not seeing more than the tokens on the current line as well as not being able to change the tokens on previous lines.
@egamma @alexandrudima

[dbaeumer]

Agree on this. I think we need to distinguish between syntax classification which happens in the render and semantic classification which happens in the language server. Syntax classification should be simple.

[thorn0]

Writing a parser for Scheme: an hour of satisfying coding
Writing a parser for JS/ES2015: years of confusion, probably still not right

— Marijn Haverbeke (@marijnjh) September 7, 2016

As the syntax of JS and TS isn't simple, its classification can't be simple. It needs a parser, you can't get by with a bunch of regexps here.

E.g. string is a type, but also it can be a name of a variable. Always using the same color for it is confusing rather than helpful. On the other hand, a name of a class always refers to the same thing both in type annotations and in expressions. Visual Studio consistently uses the same color for class names, which can be really helpful sometimes. It's just a couple of examples from the top of my head. If you need more, just see https://github.com/Microsoft/TypeScript-TmLanguage/issues

On this screenshot, VS is on the left, VS Code on the right. See how Foo gets different colors depending on whether it's a class or a variable: The highlighting is really helpful in VS, and confusing and distracting in Code.

[alexdima]

I am in favor of adding extension host API where a semantic classifier can complement a "dumb" top-down colorizer. NB even without further API this can be accomplished today through decorations, although I would not recommend it :)

[basarat]

Was annoyed by TM as well and one of the key reasons for alm. Some docs on how that was done : https://basarat.gitbooks.io/alm/content/contributing/syntax.html 🌹

[nfour]

I'm looking to interact with the TS language server in order to deliver decorations for:

• background colors based on primitive type & other stylistic crap
• decorations based on variable scope depth
• control flow decorations
• expanded interface structures on hover

Is this blocking something like what I have in mind? Or could someone point me towards resources for interacting with the TS server, such as how it provides inferred type info on hover?

[aeschli]

duplicate of #216