Error: filedescriptor out of range in select()

[zhangyafeikimi]

When my process opened too many files and fd number > 1024, this error may occur.
So, the solution is that sh.py should use poll instead of select.
All posix os have poll now.

I have a patch, however, it is based on previous sh.py. I will show it if you are interesting.

[amoffat]

@zhangyafeikimi yes, show the patch please and I will try to merge it

[zhangyafeikimi]

https://gist.github.com/zhangyafeikimi/988ccc8bd6c534b129e433f6eba9cd2e

@amoffat , It is a demo for you.
The latest sh.py has several selects, all of which should be replaced by poll.

[zhangyafeikimi]

sorry for bothering, timeout for poll should be 100, the unit is milliseconds, while it is seconds for select.

[amoffat]

Good observation @zhangyafeikimi

[zhangyafeikimi]

looking forward to the fix

[zhangyafeikimi]

@amoffat any progress? if no, need my help?

[amoffat]

@zhangyafeikimi I could use some help actually. Here's what I have so far. I'm trying to write a function, our_select that uses select.poll but behaves exactly like select.select. I know I'm not using select.poll like it was designed, but I am just trying to match the interface to select.select. For some reason, I can't get it to behave the same, and the tests fail.

One test that is interesting is FunctionalTests.test_general_signal. You can run it with:

python sh.py test FunctionalTests.test_general_signal

Anyways, it fails with an interesting error:

Testing Python2.6, locale 'en_US.UTF-8'
Exception in thread STDOUT/ERR thread for pid 6073:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/amoffat/workspace/sh/sh.py", line 1461, in wrap
    fn(*args, **kwargs)
  File "/home/amoffat/workspace/sh/sh.py", line 2384, in output_thread
    done = stream.read()
  File "/home/amoffat/workspace/sh/sh.py", line 2829, in read
    self.write_chunk(chunk)
  File "/home/amoffat/workspace/sh/sh.py", line 2804, in write_chunk
    self.should_quit = self.process_chunk(chunk)
  File "/home/amoffat/workspace/sh/sh.py", line 2725, in process
    return handler(chunk)
  File "/home/amoffat/workspace/sh/sh.py", line 1545, in fn
    return handler(chunk, *args)
  File "/home/amoffat/workspace/sh/test.py", line 1350, in agg
    process.signal(SIGINT)
  File "/home/amoffat/workspace/sh/sh.py", line 2165, in signal
    os.kill(self.pid, sig)
OSError: [Errno 3] No such process

F
======================================================================
FAIL: test_general_signal (__main__.FunctionalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/amoffat/workspace/sh/test.py", line 1357, in test_general_signal
    self.assertEqual(p, "0\n1\n2\n3\n10\n")
AssertionError: 0
1
2
 != '0\n1\n2\n3\n10\n'

----------------------------------------------------------------------
Ran 1 test in 2.515s

FAILED (failures=1)
Failed for 2.6, en_US.UTF-8

The reason why it is failing in this way is that the IO events here only occur when the process finishes, not as the subprocess writes to stdout, even though I am watching select.POLLIN on the stdout fd. You can witness this by putting print(fd, event) on the first line in the loop. You will only see events produced after the process in the test finishes.

I haven't figured it out yet, so any insight you have will be useful. I did some very shallow digging and found this bug report, which leads me to believe that select.poll may have some general problems in python... http://bugs.python.org/issue5154

[zhangyafeikimi]

Hmm, I got your code design.
You want to wrap poll as if it acts like select, maybe it is hard. All context of select need to be modified.

Don't worry about "Broken poll on MacOS", it's been fixed so far.
I will dive into the code and try to fix it.

By the way, I provide a test case for this issue:

import os
import sh

print sh.ls('.')
print sh.ls('.', '-al')

pipes = []
for i in xrange(1000):
    pipes.append(os.pipe())

# Make sure sh works fine when too many files are opened.
print sh.ls('.')
print sh.ls('.', '-al')

[amoffat]

Alright your fix is live @zhangyafeikimi. Good work