-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use socket-bounded file wrapper #58
Use socket-bounded file wrapper #58
Conversation
I expect this to break the eventlet/gevent support as makefile would be the patched makefile that does all sorts of bad things. |
I guess your actual problem was a double-close bug... can you give more details? |
Don't know how
What is double closing bug? Originally I saw that — |
Double-close bugs are situations where there would be two close() calls for the same fd. Eg, that .detach() call was supposed to prevent such problems. |
I saw these Are these double close real issues? I thought it was safe to call it many times — only first does close and others just no-op. |
Double close bugs are very dangerous imo. Worst i've seen: https://bugs.python.org/issue18748 |
Hm, that's exactly what I saw there but the opposite way. In your It's not a double close of the same file[object] (which is safe), but misused
Double close of same print('CREATESOCK')
# write(1</dev/pts/2>, "CREATESOCK\n", 11) = 11
sock, _ = socket.socketpair()
# socketpair(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0, [3<UNIX:[12057326->12057327]>, 4<UNIX:[12057327->12057326]>]) = 0
print('CLOSESOCK1')
# write(1</dev/pts/2>, "CLOSESOCK1\n", 11) = 11
sock.close()
# close(3<UNIX:[12057326->12057327]>) = 0
print('CLOSESOCK2')
# write(1</dev/pts/2>, "CLOSESOCK2\n", 11) = 11
sock.close()
# <noop>
print('DONE')
# write(1</dev/pts/2>, "DONE\n", 5) = 5
print('CREATEFILE')
# write(1</dev/pts/2>, "CREATEFILE\n", 11) = 11
file = open('/etc/hostname')
# openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_CLOEXEC) = 3</etc/hostname>
print('CLOSEFILE1')
# write(1</dev/pts/2>, "CLOSEFILE1\n", 11) = 11
file.close()
# close(3</etc/hostname>) = 0
print('CLOSEFILE2')
# write(1</dev/pts/2>, "CLOSEFILE2\n", 11) = 11
file.close()
# <noop>
print('DONE')
# write(1</dev/pts/2>, "DONE\n", 5) = 5 |
So what do you think? |
Well, this is not dangerous: This is indeed dangerous: You wouldn't happen to use python 2.7? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tested on 3.8
@@ -296,13 +294,14 @@ def handle_connection_repl(client): | |||
if _MANHOLE.redirect_stderr: | |||
patches.append(('w', ('stderr', '__stderr__'))) | |||
try: | |||
client_fd = client.fileno() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But there was no client.detach
here, only fileno
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oooooof, looks like a bug.
@@ -267,7 +265,7 @@ def exit(): | |||
raise ExitExecLoop() | |||
|
|||
client.settimeout(None) | |||
fh = os.fdopen(client.detach() if hasattr(client, 'detach') else client.fileno()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here that might be safer (until it hasattr
it)
I've tried with |
How about this change instead? @@ -265,11 +265,15 @@ def handle_connection_exec(client):
def exit():
raise ExitExecLoop()
client.settimeout(None)
- fh = os.fdopen(client.detach() if hasattr(client, 'detach') else client.fileno())
+ if hasattr(client, 'detach'):
+ client_fd = client.detach()
+ else:
+ client_fd = _ORIGINAL_DUP(client.fileno())
+ fh = _ORIGINAL_FDOPEN(client_fd)
with closing(client):
with closing(fh):
try:
payload = fh.readline()
@@ -294,11 +298,14 @@ def handle_connection_repl(client):
old_interval = getinterval()
patches = [('r', ('stdin', '__stdin__')), ('w', ('stdout', '__stdout__'))]
if _MANHOLE.redirect_stderr:
patches.append(('w', ('stderr', '__stderr__')))
try:
- client_fd = client.fileno()
+ if hasattr(client, 'detach'):
+ client_fd = client.detach()
+ else:
+ client_fd = _ORIGINAL_DUP(client.fileno())
for mode, names in patches:
for name in names:
backup.append((name, getattr(sys, name)))
setattr(sys, name, _ORIGINAL_FDOPEN(client_fd, mode, 1 if PY3 else 0))
try: |
What's the benefit of doing it this way? As I understand Why do you prefer |
There's no easy way to get an unmonkeypatched makefile afaik. If we'd follow recommended ways of doing things we wouldn't have this project in the first place, just saying :-) |
So what I would like to know is if my proposed change fixes the problem in your project. |
Haha, That's true. But it's awesome and very useful!
Is it monkeypatched at all? You mean by eventlet / gevent? If yes, it is it a problem? I've tested it and it worked fine. Do you have a case when
Maybe, need to test |
@anton-ryzhov so #60 is mostly passing. I think that pypy failure can be ignored (might be just be hitting the jit the wrong way there). Can you test that it solves your problem? |
I've tested it. Well… I don't think it solves the problem without That's how it closes the connection now (strace log):
One real close and 6 consequent redundant Another edge case — Also I still don't understand why you don't want to use |
a51043e
to
cdcf451
Compare
^ I've pulled your fixed tests into my branch |
Codecov Report
@@ Coverage Diff @@
## master #58 +/- ##
==========================================
+ Coverage 81.91% 82.72% +0.80%
==========================================
Files 6 6
Lines 1261 1279 +18
Branches 135 139 +4
==========================================
+ Hits 1033 1058 +25
+ Misses 187 182 -5
+ Partials 41 39 -2
Continue to review full report at Codecov.
|
Hmm indeed, the fdopen stuff on the stdout patching is still buggy in my changes. To be honest I'm not really sure now why I avoided makefile in the first place. Perhaps I wanted to avoid some of the crazy makefile monkeypatching in some older version of gevent/eventlet... |
Thanks for not giving up lol, I've checked now the gevent/eventlet sourcecode and it looks like nothing crazy is going on for makefile. |
Cool, thank you |
Hello,
I faced with strange behavior when used this tool — random files/connections were closed after I inspecting session. After debug I found the root cause.
os.fdopen
-created file objects only bounded to thefd
value. And they do callself.close()
on garbage collection. So if such instance is GCed after closing of original socket — it may close another file/connection that reused samefd
.Demo:
socket.makefile()
creates fileobject that is bounded to exact socket instance, so it never closes unrelatedfd
s.