Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better Idle Detection #92

Closed
benfred opened this issue Feb 16, 2019 · 4 comments
Closed

Better Idle Detection #92

benfred opened this issue Feb 16, 2019 · 4 comments
Labels
enhancement New feature or request

Comments

@benfred
Copy link
Owner

benfred commented Feb 16, 2019

The current method of marking a thread as idle is based on some heuristics - which don't work in many cases (ie, time.sleep calls can be marked as active, making it hard to view real profile data #87).

With the latest code changes to profile native extensions we have the ability to match the OS thread with the python thread id - meaning that we should be able to figure out from the OS if the thread is idle.
@benfred benfred added the enhancement New feature or request label Feb 16, 2019
@benfred benfred mentioned this issue Feb 16, 2019
Closed
benfred added a commit that referenced this issue Mar 3, 2019
@benfred
Add methods for returning thread activity
ddc3b12 
As part of #92, this adds methods
to determine if a thread is idle. This also refactors threads into a struct
with common methods across operating systems (id, active, lock etc).
@benfred benfred mentioned this issue Mar 3, 2019
Merged
@njsmith
Copy link

njsmith commented Mar 5, 2019

I'm hitting this with 0.2.0.dev1 – py-spy is telling me that my program's spending lots of time in epoll_wait, which is probably true, but not very useful :-)

@benfred benfred mentioned this issue Mar 20, 2019
Closed
benfred added a commit that referenced this issue Mar 25, 2019
@benfred
Add methods for returning thread activity
26523e3 
As part of #92, this adds methods
to determine if a thread is idle. This also refactors threads into a struct
with common methods across operating systems (id, active, lock etc).
@benfred benfred mentioned this issue Mar 30, 2019
Closed
@dimaqq
Copy link

dimaqq commented Apr 1, 2019

Let's add concurrent/futures/thread.py#L78: work_queue.get(block=True)

benfred added a commit that referenced this issue Jun 30, 2019
@benfred
Get thread activity from the OS
b5c0ac3 
Rather than use a heuristic to figure out if a thread is active, query the OS
to see if the thread is running. This should provide a more accurate view on
what's actually happening.

The tricky part here is matching the OS thread id to the python thread id,
which we were already doing for the native code unwinding. This refactors to
allow getting the native thread id even when not getting the native stack.

#92
benfred added a commit that referenced this issue Jun 30, 2019
@benfred
Get thread activity from the OS
1e58ea3 
Rather than use a heuristic to figure out if a thread is active, query the OS
to see if the thread is running. This should provide a more accurate view on
what's actually happening.

The tricky part here is matching the OS thread id to the python thread id,
which we were already doing for the native code unwinding. This refactors to
allow getting the native thread id even when not getting the native stack.

#92
benfred added a commit that referenced this issue Jun 30, 2019
@benfred
Get thread activity from the OS
4c53c95 
Rather than use a heuristic to figure out if a thread is active, query the OS
to see if the thread is running. This should provide a more accurate view on
what's actually happening.

The tricky part here is matching the OS thread id to the python thread id,
which we were already doing for the native code unwinding. This refactors to
allow getting the native thread id even when not getting the native stack.

#92
@benfred
Copy link
Owner Author

benfred commented Jul 7, 2019

This PR mostly fixes this issue #123 , by matching up the OS thread to the python thread - and then getting thread activity from the OS.

There are a couple of caveats worth pointing out:

  • we're still using the heuristic method as a fallback for x86/ARM procesesors, or if we can't get the OS threadid for any reason (on linux this only works on x86_64)
  • We get the thread activity immediately before locking the process (otherwise it's always idle), so this is a little racy

@benfred
Copy link
Owner Author

benfred commented Jul 7, 2019

Can test this out with pip install py-spy==0.2.0.dev2

@benfred benfred closed this as completed Jul 7, 2019
@benfred benfred mentioned this issue Aug 20, 2019
Closed
benfred added a commit to benfred/remoteprocess that referenced this issue Nov 17, 2019
@benfred
Add methods for returning thread activity
b4a4670 
As part of benfred/py-spy#92, this adds methods
to determine if a thread is idle. This also refactors threads into a struct
with common methods across operating systems (id, active, lock etc).
@synapticarbors synapticarbors mentioned this issue Aug 6, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benfred @njsmith @dimaqq