Support 3.10's new line number API

[itamarst]

Python 3.10 and later have a new, and supposedly more accurate, scheme for mapping the bytecode index to line number, deprecating co_lnotab.

Details here: https://www.python.org/dev/peps/pep-0626/, and py-spy specific details here: https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers

To expand on the latter, it seems py-spy is encouraged to include the traversal C API by copy/pasting the relevant C code (either via cc crate or via translating to Rust): https://github.com/python/cpython/blob/e9cd47d0e58cd468d6482d7ba59730b134d0d521/Objects/codeobject.c#L685

(It's supposed to be standalone code that doesn't depend on the rest of CPython code base.)

Options appear to be:

1. cc + copy/pasting C code for each major post-3.10 release.
2. Parse new format directly w/Rust code. https://github.com/python/cpython/blob/cc8ecf6864375994899fd79d601ab05d0df9edbf/Objects/lnotab_notes.txt documents the new format. Again might need to write new parser for e.g. 3.11, format is not guaranteed to be stable.
3. Stick to old co_lnotab format, but it'll be removed eventually, and also PEP says it's populated "lazily" so possibly it's not actually available, would need to check the code.

[benfred]

Thanks for the reminder!

This is something we need to get in before 3.10 is released - which is currently scheduled for October 4th: https://www.python.org/dev/peps/pep-0619/ . I agree with the options you presented - I think we will do either option 1 or 2.

I've started working on python 3.10 support here #442 . It still needs the line number api changes, but everything else seems like its working right now.

For reference, there was a conversation about how this line number api would affect py-spy last year python/peps#1555 - and the python core devs were pretty awesome in considering how this change would affect py-spy.

[benfred]

(btw - filprofiler is super cool =)

[itamarst]

The nascent commercial variant of Fil is my ulterior motivation here because I need the same algorithms, just with PyO3. If there's a way to support both your API and PyO3 API in same codebase, would be happy to co-maintain a crate.

I should mention of course that pyspy has been quite useful to me, e.g. "this integration test deadlocks on windows" went from utterly mystifying to trivial to fix by just attaching py-spy, seeing as I have no idea how to get a Python stack trace with Windows toolchains, or even how to use a debugger.

(There's another conversation I want to have at some point about starting a little community with the purpose of both talking about shared interests, and exposing the rather specialized knowledge needed for profilers and the like to a broader set of people.)

[itamarst]

I guess technically I won't be using PyO3, I'll be using just the bytes with the compressed index, so seems like it ought to be pretty simple to have a function per-Python version that takes that, bytecode index (unmodified) from code object, and returns the line number.

(Or if it's easier, there's always copy/paste with credit).

[benfred]

I've added support for python 3.10 here (including the line number API) https://github.com/benfred/py-spy/pull/442/files and python 3.10 support will be in the py-spy version v0.3.10 - which I'm pushing out now.

@itamarst Glad to hear py-spy has been useful - and hopefully some of this code will come in useful too. When you mention I need the same algorithms, just with PyO3, do you mean that you need to also decode line number tables for different versions of python, but from python instead of rust?

I don't have python bindings for py-spy yet, but did do a basic exploration several years ago https://www.benfrederickson.com/writing-python-extensions-in-rust-using-pyo3/ . I think it shouldn't be too hard to add python bindings with pyO3 - I'd probably go with maturin now for building instead of the CI approach I laid out there, but otherwise I think the approach is still similar.

Fwiw, It's currently possible to use py-spy as a rust library (instead of a CLI application), and there is an example here https://github.com/benfred/py-spy/blob/master/examples/dump_traces.rs . If you just want to decode a python line number table, you'll have to dig into the internals a bit though - and I don't think the underlying API is public yet =(.

I'd also be interested in talking more about python profiling =) . Let me know if you end up re-implementing any of this code - I might be able to save you some time (for instance, the one tricky thing with the new line number table support was that the 'lasti' values need doubled - which took me way too long to figure out this weekend:

py-spy/src/python_interpreters.rs

Lines 245 to 249 in 913e95c

	// in Python 3.10 we need to double the lasti instruction value here (and no I don't know why)
	// https://github.com/python/cpython/blob/7b88f63e1dd4006b1a08b9c9f087dd13449ecc76/Python/ceval.c#L5999
	// Whereas in python versions up to 3.9 we didn't.
	// https://github.com/python/cpython/blob/3.9/Python/ceval.c#L4713-L4714
	let lasti = lasti * 2;

)

[itamarst]

Thank you!

Good to know py-spy can be used as a crate, I'll probably submit a PR then when I reach the point where I need the APIs (I'm accessing Python APIs via PyO3, so when I decode line numbers it'll be in Rust, but with the difference that I get the bytes for the line<->bytecode mapping records via PyO3 instead of memory mapping).

I had an idea for an UX improvement for performance flamegraphs which I am trying to prototype, BTW, involving including whether each stack is CPU/idle/blocked on IO, via process status flags (e.g. https://psutil.readthedocs.io/en/latest/#psutil.Process.status). On Linux at least where in practice each thread is (from kernel perspective) a process, you can get this for all threads. Will share if I manage a working prototype.