Token lines in the tokenize module always end in new line

[pablogsal]

As a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.

Linked PRs

• gh-104825: Remove implicit newline in the line attribute in tokens emitted in the tokenize module #104846
• [3.12] gh-104825: Remove implicit newline in the line attribute in tokens emitted in the tokenize module (GH-104846). #104850
• [3.12] gh-104825: add omitted idlelib text fix #104880

[lysnikolaou]

Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?

[pablogsal]

Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?

I am working on a fix, should be up in half an hour or so

[lysnikolaou]

#104846 now removes a newline, even when a newline exists in the original source.

❯ ./python.exe
Python 3.13.0a0 (heads/setuptools-peg-generator-dirty:c7503656c2, May 24 2023, 12:24:20) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> import io
>>> file = io.StringIO('1 + 2 + 3\n')
>>> for token in tokenize.generate_tokens(file.readline):
...     print(token)
... 
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 2), end=(1, 3), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='2', start=(1, 4), end=(1, 5), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 6), end=(1, 7), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='3', start=(1, 8), end=(1, 9), line='1 + 2 + 3')
TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 9), end=(1, 10), line='1 + 2 + 3')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')

[lysnikolaou]

@pablogsal Do you think the above should be fixed? Will you work on it?

[pablogsal]

@pablogsal Do you think the above should be fixed? Will you work on it?

Hummmmm, not sure how easy is going to be for us to distinguish between implicit and explicit newlines at the end. I would say, leave it like is now unless you disagree, in which case let's see if we can fix it easily.

[lysnikolaou]

My only concern is that tests that rely on that will fail. For example, test_pegen now fails:

def test_repeat_0_simple(self) -> None:
        grammar = """
        start: thing thing* NEWLINE
        thing: NUMBER
        """
        parser_class = make_parser(grammar)
        node = parse_string("1 2 3\n", parser_class)
        self.assertEqual(
            node,
            [
                TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1 2 3\n"),
                [
                    TokenInfo(
                        NUMBER, string="2", start=(1, 2), end=(1, 3), line="1 2 3\n"
                    ),
                    TokenInfo(
                        NUMBER, string="3", start=(1, 4), end=(1, 5), line="1 2 3\n"
                    ),
                ],
                TokenInfo(
                    NEWLINE, string="\n", start=(1, 5), end=(1, 6), line="1 2 3\n"
                ),
            ],
        )
        node = parse_string("1\n", parser_class)
        self.assertEqual(
            node,
            [
                TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1\n"),
                [],
                TokenInfo(NEWLINE, string="\n", start=(1, 1), end=(1, 2), line="1\n"),
            ],
        )

The failure is:

======================================================================
FAIL: test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/lysnikolaou/repos/python/cpython/Lib/test/test_peg_generator/test_pegen.py", line 248, in test_repeat_0_simple
    self.assertEqual(
AssertionError: Lists differ: [Toke[67 chars]1 2 3'), [TokenInfo(type=2 (NUMBER), string='2[201 chars] 3')] != [Toke[67 chars]1 2 3\n'), [TokenInfo(type=2 (NUMBER), string=[209 chars]\n')]

First differing element 0:
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3')
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3\n')

Diff is 1018 characters long. Set self.maxDiff to None to see it.

The fix is very simple, so I'm okay with going with this as-is. I will fix test_pegen in my setuptools PR.

[nedbat]

You say:

As a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.

But it looks to me like the lines used to always end in newline, except for ENDMARKER. This uses tokbug.py from #104972:

3.10.9 (main, Dec  7 2022, 07:15:24) [Clang 12.0.0 (clang-1200.0.32.29)]
TokenInfo(type=62 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='b\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')