3.10.0b2 traces with-exit before the break that caused the exit

[nedbat]

BPO	44298
Nosy	@nedbat, @markshannon, @pablogsal
PRs	bpo-44298: Fix line numbers for early exits in with statements. #26513 [3.10] bpo-44298: Backport #26513 to 3.10 #26516

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/markshannon'
closed_at = <Date 2021-06-03.19:59:00.106>
created_at = <Date 2021-06-03.10:05:02.392>
labels = ['interpreter-core', 'type-bug', '3.10', 'release-blocker']
title = '3.10.0b2 traces with-exit before the break that caused the exit'
updated_at = <Date 2021-06-03.21:41:17.713>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2021-06-03.21:41:17.713>
actor = 'nedbat'
assignee = 'Mark.Shannon'
closed = True
closed_date = <Date 2021-06-03.19:59:00.106>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-06-03.10:05:02.392>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 44298
keywords = ['patch', '3.10regression']
message_count = 8.0
messages = ['394990', '394991', '394994', '394995', '395007', '395033', '395038', '395052']
nosy_count = 3.0
nosy_names = ['nedbat', 'Mark.Shannon', 'pablogsal']
pr_nums = ['26513', '26516']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44298'
versions = ['Python 3.10']

[nedbat]

Python 3.10 now traces back to with statements when exiting the with block. When the exit is a break statement, the with exit is visited before the break statement is. This seems confusing.

--- 8< -----------------------------------------

import linecache, sys

def trace(frame, event, arg):
    # The weird globals here is to avoid a NameError on shutdown...
    if frame.f_code.co_filename == globals().get("__file__"):
        lineno = frame.f_lineno
        print("{} {}: {}".format(event[:4], lineno, linecache.getline(__file__, lineno).rstrip()))
    return trace

def doit():
    for i in range(2):
        with open("test", "w") as f:
            a = 13
            b = 14
            break
    c = 16

print(sys.version)
sys.settrace(trace)
doit()

--- 8< -----------------------------------------

3.10.0b2 (default, Jun 3 2021, 05:27:13) [Clang 12.0.0 (clang-1200.0.32.29)]
call 10: def doit():
line 11: for i in range(2):
line 12: with open("test", "w") as f:
line 13: a = 13
line 14: b = 14
line 12: with open("test", "w") as f:
line 15: break
line 16: c = 16
retu 16: c = 16

Shouldn't we get a trace for line 15 (break), then line 12 (with-exit), then line 15 again, then line 16?

[nedbat]

(I'm not sure whether to create other issues for further details)

I'm also seeing a return in a with will trace withexit/return for a plain "return" statement, but return/withexit/return for a return with a value ("return 17").

I would expect that a statement causing an exit from a with block would always be traced before the with-exit.

[markshannon]

For context, this behavior was introduced in https://bugs.python.org/issue43933

[markshannon]

Why this occurs:

with cm: 
    A
    break

translates to something like:

ex = cm.__exit__; cm.__enter__()  # with cm
    A
    ex(...)
    goto loop_end   # break

So, the break is traced after the exit call.

However, this doesn't seem consistent with try-finally statements which trace any break/continue/return before the finally block.

[markshannon]

New changeset 937cebc by Mark Shannon in branch 'main':
bpo-44298: Fix line numbers for early exits in with statements. (GH-26513)
937cebc

[pablogsal]

New changeset cea0585 by Mark Shannon in branch '3.10':
[3.10] bpo-44298: Backport bpo-26513 to 3.10 (bpo-26516)
cea0585

[pablogsal]

Ned, can you confirm that this works for you? I am closing the issue, but if something is missing, please, reopen it and we will look into it :)

[nedbat]

Thanks for the quick turnaround, this works!

[gpshead]

this appears to have caused a regression in line numbers for return statements. #93975