Failure in test_string_literals: Unexpected amount of warnings

[lysnikolaou]

Current Parser:

╰─ ./python       
Python 3.9.0a5+ (heads/pegen:99a8e2fa08, Apr  9 2020, 16:19:04) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> with warnings.catch_warnings(record=True) as w:
...     warnings.simplefilter('always', category=DeprecationWarning)
...     eval("'''\n\\z'''")
... 
'\n\\z'
>>> len(w)
1

Pegen:

╰─ ./python -p new
Python 3.9.0a5+ (heads/pegen:99a8e2fa08, Apr  9 2020, 16:19:04) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> with warnings.catch_warnings(record=True) as w:
...     warnings.simplefilter('always', category=DeprecationWarning)
...     eval("'''\n\\z'''")
... 
'\n\\z'
>>> len(w)
2

[gvanrossum]

Simpler repro (the eval is just confusing matters):

Old parser:

>>> '\z'
<stdin>:1: DeprecationWarning: invalid escape sequence \z
'\\z'
>>> 

New parser:

>>> '\z'
<stdin>:1: DeprecationWarning: invalid escape sequence \z
<stdin>:1: DeprecationWarning: invalid escape sequence \z
<stdin>:1: DeprecationWarning: invalid escape sequence \z
'\\z'
>>> 

[lysnikolaou]

If we memoize the atom rule, this goes away, since this is only happening, because the parser tries to parse a string more than once, before it completely fails and it backtracks out of everything.

Should we do it?

[lysnikolaou]

It seems that there is another mismatch here, that I had missed.

Current Parser:

>>> with warnings.catch_warnings(record=True) as w:
...     eval("'''\n\\z'''")
... 
'\n\\z'
>>> w[0].lineno
1

Pegen:

>>> with warnings.catch_warnings(record=True) as w:
...     eval("'''\n\\z'''")
... 
'\n\\z'
>>> w[0].lineno
2

Not sure which is wrong.

[gvanrossum]

Neither points to the line with the offending \z on it -- the old parser points to the start of the string, the new parser to the end. Let's say that the old parser is right then.

[lysnikolaou]

Reopening this until we disable the line checking test.

[lysnikolaou]

Turns out pegen does not point to the end of the string, but to the ENDMARKER token here. The problem is that STRING nodes are parsed in a loop, which means that p->fill - 1 in
https://github.com/python/cpython/blob/531d1e541284bfd7944f8c66a5e8c3c3234afaff/Parser/pegen/pegen.c#L389 will always point to the node after a STRING, which I think is a bug and we need to find a way to resolve.