Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.12 support #633

Closed
kakkoyun opened this issue Nov 14, 2023 · 13 comments
Closed

Python 3.12 support #633

kakkoyun opened this issue Nov 14, 2023 · 13 comments

Comments

@kakkoyun
Copy link

kakkoyun commented Nov 14, 2023

Hey @benfred, first of all, I'm a massive fan of the project! I have learned a lot about Python by reading this codebase [1].

I have seen that Python 3.12 tests are failing #618
and also, starting from 3.12 3.13, Python provides a struct with offsets for out-of-process tools python/cpython#106598

I would like to ask about the state (or plan) for the Python 3.12 support and offer help if you would like to accept contributions for this part.

[1]: As a testament to that, I have created this project to use eBPF for profiling Python code.

@kakkoyun
Copy link
Author

and also, starting from 3.12 3.13, Python provides a struct with offsets for out-of-process tools python/cpython#106598

I was overly enthusiastic about the changes. Turns out they're merged into the main and scheduled to be released for the 3.13 release.

 git branch -r --contains b444bfb0a325dea8c29f7b1828233b00fbf4a1cb
  origin/HEAD -> origin/main
  origin/main

Regardless, I'm still eager to help with 3.12 support.

@kakkoyun
Copy link
Author

Relevant: _PyRuntime.tstate_current removed for the sake of storing it in a thread-local storage python/cpython#103323

@benfred
Copy link
Owner

benfred commented Nov 17, 2023

Thanks for the links @kakkoyun ! I've started on this - and have a very basic implementation running locally. There are currently a couple of issues that I still need to sort out on this though before we can fully support python 3.12:

  • GIL state has been moved to thread local storage like you mentioned - and detecting which thread holds the GIL no longer works
  • Thread names aren't getting picked up
  • Line numbers are incorrect
  • There seems to be a new <interpreter trampoline> frame in the stack - that we might want to filter out

As an example here is the current py-spy dump output on the tests/scripts/longsleep.py test case:

Process 43243: python tests/scripts/longsleep.py
Python v3.12.0 (/home/ben/miniconda3/envs/py12/bin/python3.12)
Thread 43243 (idle)
    longsleep (longsleep.py:-359)
    <module> (longsleep.py:-3)
    <interpreter trampoline> (<shim>:1)

You can see most of the issues here (aside from the missing GIL) - the line numbers are clearly wrong, its not picking up the thread name etc , but aside from that its at least starting to get the right output.

I'm hoping to get everything aside from the GIL detection working in the next couple of days, and get a release out with this (and some other fixes).

@kakkoyun
Copy link
Author

Thanks for the detailed update, @benfred. Please don't hesitate to let me know if I can be helpful.

I can try to take a shot at the GIL detection problem.

@benfred
Copy link
Owner

benfred commented Nov 21, 2023

Initial PR is up here #642 - still needs a bunch of work (including line numbers being wrong), but its a start at least

@mdczaplicki
Copy link

Hey there, any update on this?

@ianozsvald
Copy link

I'm working on our 3rd edition of O'Reilly's High Performance Python book. I've covered Py-Spy in the previous edition and I will continue to show it. We're using Python 3.12 for the book (it'll still be relevant when we publish early in 2025). Currently I'm noting that Py-Spy doesn't support 3.12, it'd be great to know if there's a timeline for support. I'd happily change that note in the book once 3.12 is supported. Thanks for your continued support on this tool, it can be very handy!

@kakkoyun
Copy link
Author

@benfred I finally figured out how to read the thread state from the TLS. (i.e. parca-dev/parca-agent#2553)

I hope to find some free cycles in the upcoming weeks to port changes to here.

@mg-christian-axelsson
Copy link

Any progress on this issue?

@kakkoyun
Copy link
Author

I haven't started it yet. I recently changed jobs, and I haven't been able to find any free-cycles yet. If anyone else has more time, I'd be happy to help with the reviews and guidance.

@battaglia01
Copy link

What is left to do? Just the line numbers?

@olejorgenb
Copy link

olejorgenb commented Aug 8, 2024

Since there's so many duplicates I'll risk "spamming" this issue with the alternatives I posted here (TIL: use https://github.com/P403n1x87/austin for now)

@zanieb
Copy link
Contributor

zanieb commented Oct 15, 2024

I believe this is closed by #642

@benfred benfred closed this as completed Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants