Always 100% cpu usage using _bg=True

[rw]

If I run this script using the latest code from master (703657b):

import time

import sh

x = sh.sleep(60, _bg=True)
time.sleep(60)
x.wait()

CPU usage spikes to 100% and stays there for the duration of the command. Is there a way around this bug?

OSX 10.11.6, Pythons 2.7.12 and 3.5.2

[amoffat]

shhh...mining bitcoins 💰

thanks for reporting. just pushed up 1.12.4 live with the regression fix. should be fine now. close the issue if it looks good

[rw]

Much better! It's still eating up CPU though. About 5% CPU per waiting process, so when I run this it eats 20% CPU:

import time

import sh

procs = []
for i in range(4):
    procs.append(sh.sleep(60, _bg=True))
time.sleep(60)

[amoffat]

hm..not as high on my machine. i just inserted a sleep into a freely-spinning background thread that has to run. let me take a deeper look tomorrow

[rw]

I'll close this issue since it's looking a lot better, but I do think someone should examine it further. Thanks!

[amoffat]

made some more improvements in release 1.12.5 and got 100 processes down to ~5% cpu on my machine.

import sh

procs = []                                                                                                        
for i in range(100):                                                                                              
    p = sh.sleep(30, _bg=True)                                                                                   
    procs.append(p)                                                                                                                                                                                                                  
[p.wait() for p in procs]

want to have one more look and confirm @rw ?

[rw]

Looks a lot better, about 10-20% on my machine for those 100 processes.

Is there an evented way to do wait? It seems like a waste to be polling so much.

[amoffat]

I know what you mean. I was able to replace most of the internal polling with threading.Events, but in some places, mostly related to timeouts, we have to remain responsive. For example, there's a thread that runs select.select on the process's output pipe/tty. That thread also needs to check if the timeout Event.is_set(), so we can't wait indefinitely on select.select...we can only select for a very short time. The speed that that thread spins is directly tied with how fast the process will finish and clean up. But the speed of that thread spinning contributes to the CPU

[rw]

Ah okay. Is there an ergonomic way to let the user configure how much load this puts? E.g. if I'm running 100 procs that I know will take about 10 minutes to run, should I be able to increase the polling interval?

[amoffat]

No, there's not. The closest you would have to something like that would be to use _fg=True, which will spawn the process in such a way that there are no background threads, since the input/outputs will be directly connected (via os.dup2) to sys.stdin/out/err. You would see much lower CPU usage there, but you will be unable to capture the output.

[amoffat]

But you also won't be able to batch-launch processes with _fg=True, so that probably doesn't work for you.

[rw]

I suggest adding a caveat about CPU usage to the documentation for _bg=True.