lib2to3 grammar no longer able to parse Python 3.10.

[qwattash]

It appears that with the introduction of PEP 634 the lib2to3 parser is no longer able to correctly parse all python code.
Also from the 2to3 documentation:

Deprecated since version 3.11, will be removed in version 3.13: Python 3.9 switched to a PEG parser (see PEP 617) while lib2to3 is using a less flexible LL(1) parser. Python 3.10 includes new language syntax that is not parsable by lib2to3’s LL(1) parser (see PEP 634). The lib2to3 module was marked pending for deprecation in Python 3.9 (raising PendingDeprecationWarning on import) and fully deprecated in Python 3.11 (raising DeprecationWarning). It will be removed from the standard library in Python 3.13. Consider third-party alternatives such as LibCST or parso.

I'm not sure what the correct course of action might be.

[kamahen]

See also Issue #894, Issue #448, and possibly other issues. (A number of alternatives are discussed - see also psf/black#2318)
I think that there's an intermediate solution, using a thin layer over the builtin ast module.

[isanbard]

I started creating a new parser that uses the ast and parser modules from Python. I haven't had a chance to hook it in completely, because a lot of code exists around tweaking the split penalties—virtually all of which needs to be cleaned up. BUT it has potential to support Python going into the future.

The new code is in top-of-tree now, if you'd like to take a look.

[kamahen]

Is the new parser intended to become a separate package, or only used by yapf?
(Reason for asking: I could use a whitepace-aware parser replacement for lib2to3 for something I'm working on)

[isanbard]

I would be happy to make it a separate package. It might take a bit, because of lack of time etc. So feel free to crib parts of it from YAPF.

[e-gebes]

Hello,
is there an update on the progress of this, do you think you will succeed with supporting the new Python 3.10 syntax?

Thank you for an update and your efforts!

[bwendling]

Closed with 7c408b9.

[jni]

Amazing! Thank you!!! 🙏

[cameron-simpson]

Thanks from me as well!