Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(3.12+) tokenize exception type is wrong #105390

Closed
asottile opened this issue Jun 6, 2023 · 1 comment
Closed

(3.12+) tokenize exception type is wrong #105390

asottile opened this issue Jun 6, 2023 · 1 comment
Assignees
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error

Comments

@asottile
Copy link
Contributor

asottile commented Jun 6, 2023

Bug report

the document exception type for the tokenize.* functions is TokenError -- however it appears it now raises SyntaxError: https://docs.python.org/3.12/library/tokenize.html#tokenize.TokenError

Raised when either a docstring or expression that may be split over several lines is not completed anywhere in the file, for example:

import io
import tokenize

list(tokenize.generate_tokens(io.StringIO('"""').readline))
$ python3.11 t.py
Traceback (most recent call last):
  File "/home/asottile/workspace/switch-microcontroller/t.py", line 4, in <module>
    list(tokenize.generate_tokens(io.StringIO('"""').readline))
  File "/usr/lib/python3.11/tokenize.py", line 465, in _tokenize
    raise TokenError("EOF in multi-line string", strstart)
tokenize.TokenError: ('EOF in multi-line string', (1, 0))

and

#  python3.13 t.py
Traceback (most recent call last):
  File "/y/pyupgrade/t.py", line 4, in <module>
    list(tokenize.generate_tokens(io.StringIO('"""').readline))
  File "/usr/lib/python3.13/tokenize.py", line 526, in _generate_tokens_from_c_tokenizer
    for info in it:
  File "<string>", line 1
    """
    ^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)

Your environment

  • CPython versions tested on: c7bf74b
  • Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs

@asottile asottile added the type-bug An unexpected behavior, bug, or error label Jun 6, 2023
@pablogsal pablogsal self-assigned this Jun 6, 2023
@pablogsal
Copy link
Member

pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2023
pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2023
@AlexWaygood AlexWaygood added 3.12 bugs and security fixes 3.13 bugs and security fixes labels Jun 6, 2023
pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2023
pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 7, 2023
…r tokenize errors (pythonGH-105399)

(cherry picked from commit ffd2654)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
pablogsal pushed a commit that referenced this issue Jun 7, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 7, 2023
(cherry picked from commit 264a011)

Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants