Fix line numbers generated by the compiler

[iritkatriel]

The line number calculation is a little off in a few places, and involves some guesswork and estimation. This issue is to make it more accurate, so that we can safely move this code to before the optimization stage (see python/cpython#87092).

Particular issues:
python/cpython#98390
python/cpython#98398
python/cpython#98442
python/cpython#98461
python/cpython#98762

[iritkatriel]

Question for @markshannon re python/cpython@c544093 .

Where in PEP 626 is it prescribed that the POP_TOP after an expression statement is artificial? Could it not be assigned the location of the expression? (I don't see any discussion of this in python/cpython#86412).

[iritkatriel]

The end result (after propagating line numbers etc) is actually the same as what we would get if we do what I suggest above. So maybe it's ok to just assign that lineno at codegen time?

def g():
  f(1)
  f(2)
  f(3)

import dis
from pprint import pprint
pprint(list([(i.opname, i.positions) for i in dis.get_instructions(g)]))

Output:

[('RESUME', Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0)),
 ('LOAD_GLOBAL',
  Positions(lineno=2, end_lineno=2, col_offset=2, end_col_offset=3)),
 ('LOAD_CONST',
  Positions(lineno=2, end_lineno=2, col_offset=4, end_col_offset=5)),
 ('CALL', Positions(lineno=2, end_lineno=2, col_offset=2, end_col_offset=6)),
 ('POP_TOP', Positions(lineno=2, end_lineno=2, col_offset=2, end_col_offset=6)),
 ('LOAD_GLOBAL',
  Positions(lineno=3, end_lineno=3, col_offset=2, end_col_offset=3)),
 ('LOAD_CONST',
  Positions(lineno=3, end_lineno=3, col_offset=4, end_col_offset=5)),
 ('CALL', Positions(lineno=3, end_lineno=3, col_offset=2, end_col_offset=6)),
 ('POP_TOP', Positions(lineno=3, end_lineno=3, col_offset=2, end_col_offset=6)),
 ('LOAD_GLOBAL',
  Positions(lineno=4, end_lineno=4, col_offset=2, end_col_offset=3)),
 ('LOAD_CONST',
  Positions(lineno=4, end_lineno=4, col_offset=4, end_col_offset=5)),
 ('CALL', Positions(lineno=4, end_lineno=4, col_offset=2, end_col_offset=6)),
 ('POP_TOP', Positions(lineno=4, end_lineno=4, col_offset=2, end_col_offset=6)),
 ('LOAD_CONST',
  Positions(lineno=4, end_lineno=4, col_offset=2, end_col_offset=6)),
 ('RETURN_VALUE',
  Positions(lineno=4, end_lineno=4, col_offset=2, end_col_offset=6))]

[iritkatriel]

Ah, I see now (from the test in the commit I linked above) why this was done. The expression extends beyond the function call.

[iritkatriel]

I've done as much as I intend to do on this - the line-number mechanism is more under control now, the individual cases I didn't cover will be done separately as we find issues.