Support PEP 701: Syntactic formalization of f-strings

[dhruvmanila]

PEP 701 formalizes the grammar for f-strings. This will require updates to both the lexer and parser while keeping the AST representation the same.

• Update the lexer to emit the new tokens (FSTRING_START, FSTRING_MIDDLE, FSTRING_END). This means that f"foo {bar}" wouldn't be emitted as a single String token but rather something like:

   FSTRING_START('f"')
   FSTRING_MIDDLE("foo ")
   LBRACE
   NAME("bar")
   RBRACE
   FSTRING_END('f"')
  

• Update the parser to account for any grammar changes required to recognize these tokens and parse it into an equivalent AST nodes. There are few things to note here:

  • We can remove the f-string parsing done in the string parser (string.rs)
  • Debug expressions (f"foo {bar = }") preserves whitespaces using the debug_text field. Newlines needs to be preserved as well.
  • Implicit string concatenation
  • SimpleTokenizer changes

The goal is to complete this in the current iteration (August) before the official release of Python 3.12.

Tasks

• Add support for PEP 701 in the lexer #7042
• Add support for PEP 701 in the parser #7043
• Split/Remove Backward tokenization #6363
• PEP 701 linter changes #7299
• Update FormatStringContinuation to use the new f-string tokens #7517

Reference implementation:

• CPython: Tokenizer
• LibCST: Tokenizer | Parser

[dhruvmanila]

The way the PRs will be merged is as follows:

1. All changes will be made in separate pull request for separation in logic and code reviews
2. All the pull requests will be merged in a new branch dhruv/pep-701
3. A final pull request will be opened from dhruv/pep-701 to main once changes are done.
4. Merge 🥳