Tokenization freezes with long line like "((a+a) (a+a) (a+a) (a+a) ...

[alexr00]

From @Maxpxt in microsoft/vscode#89846

Details

What editor are you seeing the problem in? (e.g. Atom, Visual Studio Code, etc.)
VS Coe
What version of the editor are you using?
VS Code version: Code 1.41.1 (26076a4, 2019-12-18T14:58:56.166Z)
What color theme are you using?
Not applicable

Repro

Create a file in plain text mode and fill it with:
((a+a) + ~30 times  (a+a), which looks like ((a+a) (a+a) (a+a) (a+a) ...

Change the file language to C#.

This causes VSCode to stop responding.

This does not happen with other languages like C++, F#, Java, JavaScript.

Interestingly, in the following slight variations of this, everything works fine.

( on line 1 + (a+a) (a+a) (a+a) (a+a) ... on line 2
((a+a)(a+a)(a+a)(a+a) ...
When the line is smaller (around 15 to 20 times  (a+a)), the the editor does not stop responding, but takes long to respond to any changes made to the file (like typing a letter). The larger the line, the longer it takes.