Unicode variable names crash yapf

[MrYann]

Minimal working example :

def test(x):
	Δ=x+2
	print(Δ)

test(4)

this is legal python3, but yapf breaks and does nothing on a file with unicode variables. As can be seen here: the missing spaces in Δ=x+2 were not added.

The error comes from lib2to3 (lib2to3.pgen2.parse.ParseError: bad token: type=58, value='Δ', context=('', (2, 1))) but I can not say if it is a bug in 2to3, or in yapf.

[audiolion]

yapf really doesn't support python 3 fully, just what lib2to3 can handle, and this error is explicitly coming from lib2to3.

[MrYann]

It is. But, perhaps, yapf could handle the exception in some less extreme way ? Also, could you give a hint as to what i should open as an issue in lib2to3 ?

…

_____ Yann

Le 4 mai 2017 à 21:57, Ryan Castner ***@***.***> a écrit : yapf really doesn't support python 3 fully, just what lib2to3 can handle, and this error is explicitly coming from lib2to3. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[audiolion]

I just wanted to let you know that the author of yapf is a googler, google's python codebase is almost all py 2.7 and so there is little interest in developing tools or supporting python 3, so besides what can be gotten from lib2to3 there likely won't be much effort here

[bwendling]

@MrYann You can add a bug report here: http://bugs.python.org/

[MrYann]

Ok. But what should i report ? My question is: is it lib2to3 that is not behaving as expected, or is it yapf that is not handling the exception as it should ? Independantly of the version of Python, being robust to unexpected symbols could be of interest.

…

_____ Yann

Le 7 mai 2017 à 10:01, Bill Wendling ***@***.***> a écrit : @MrYann You can add a bug report here: http://bugs.python.org/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[bastih]

@MrYann it's lib2to3 not being completely up-to-date with the latest python version. So it's their bug.

[MrYann]

I opened an issue regarding lib2to3.parse.classify at http://bugs.python.org/. Their reply was:

"Why? Unicode identifiers are not allowed in 2.x. I don't think lib2to3 is able or responsible to parse invalid syntax source codes"

So, maybe, could it be possible to patch the grammar invoked during the "python3" parsing so that parse.ParseError would not trigger in this case ?

[audiolion]

hmm, that does make sense, their library is for going from 2 to 3, not accepting 3 constructs, looks like yapf might have to ignore this error as you stated

[bwendling]

I find their reasoning to be a bit disingenuous. They should certainly know that the lib2to3 parser is used outside of the 2to3 utility. Besides, if a Python file is only partially converted to Python 3, shouldn't the 2to3 utility still be able to convert any left over Python 2-isms?

[tttapa]

This still appears to be an issue in the latest version, and it looks like both the 2to3 parser and the yapf error handling fail.

For example:

h̅ = 1.054571817e-34 # Js (Planck's constant over 2π)

(h̅ is U+0068 U+0305)

Yapf output:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 119, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/parse.py", line 119, in addtoken
    ilabel = self.classify(type, value, context)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/parse.py", line 175, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='̅', context=('', (1, 1))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 183, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 125, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/parse.py", line 119, in addtoken
    ilabel = self.classify(type, value, context)
  File "/home/user/.local/lib/python3.10/lib2to3/pgen2/parse.py", line 175, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='̅', context=('', (1, 1))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__init__.py", line 225, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 96, in FormatFile
    reformatted_source, changed = FormatCode(
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 186, in FormatCode
    raise errors.YapfError(errors.FormatErrorMsg(e))
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/.local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__main__.py", line 18, in <module>
    yapf.run_main()
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__init__.py", line 360, in run_main
    sys.exit(main(sys.argv))
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__init__.py", line 124, in main
    changed = FormatFiles(
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__init__.py", line 202, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "/home/user/.local/lib/python3.10/site-packages/yapf/__init__.py", line 236, in _FormatFile
    raise errors.YapfError(errors.FormatErrorMsg(e))
  File "/home/user/.local/lib/python3.10/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
IndexError: tuple index out of range