Non-ASCII identifiers not supported

[p-avital]

Ever since PEP3131 (Python 3.0), non-ASCII characters are supported in identifiers (though it hasn't become a very popular feature, most people don't know that's the case). However, Pylance considers them a syntax error:

def plt_Γsplit(path):
    data: List[Dict] = parse_csv(path)
    ...

Shows multiple errors, the first of which being that ( is expected at the position where Γ is.

Any chance of a quick fix on that ? :)

[erictraut]

Pylance already has full support for PEP3131. I'm not able to repro a problem with the code you posted above. Are you sure you do not have some other language server (like MPLS or Jedi) also enabled?

[erictraut]

Update... I was able to repro the problem. It happens sporadically, which is very odd. I'll work on getting a consistent repro and then dig into the problem.

[p-avital]

Thanks for the update, I was kinda weirded out considering I was starting from what I believe to be a clean config.
Btw very efficient issue handling, thanks for that too :)

[erictraut]

I was able to find a consistent repro and track down the bug. The tokenizer in pylance/pyright lazily initializes the Unicode character tables that define which characters can be used legally within identifiers. It was missing this lazy initialization in one code path.

The bug is now fixed, but it's probably too late to get it into this week's pylance release. It will be in next week's release.

Thanks for reporting the problem!

[p-avital]

I did not expect such reaction speeds, thanks. If you have a trick to make it work meanwhile, it'd help (and I always find tricks around edge cases fun so I'm curious).
Should I close the issue?

[erictraut]

Please leave it open for now. We'll close it once we release the fix.

If the first character in your identifier is a non-ASCII character, it will properly initialize the Unicode tables. All of my unit tests were constructed this way, which is why I missed this case.

If you're curious about the change, it's here.

[jakebailey]

Yeah, it didn't quite make it into this release. Next week, definitely.

[jakebailey]

This issue has been fixed in version 2020.10.1, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#2020101-14-october-2020