-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Werkzeug crashes after writing to closed pipe #954
Comments
Use a production WSGI server such as Gunicorn or uWSGI, not the Werkzeug dev server. |
I have a very similar problem except that I'm using the Werkzeug dev server for development (which is, AFAICT, its intended use), i.e with the browser directly connected to port 5000. The error happens several times per hour, forcing a manual restart of the server to keep developing. Here's a traceback:
|
I experience the very same issue as @sfermigier with development server ( |
This behavior happens usually in a very simple case: you're using some sort of auto-complete features. The browser starts a connection for query tokens and then stops and starts another request for more tokens. You'll end up with lots of Broken Pipes. And this wasn't a problem until last release which makes the dev server completely blocked. |
If it's related, I can confirm I experience this issue using |
Does anyone have solution that doesn't include running a WSGI server? I seem to be running into this problem with bots performing a SYN scan against my host. |
@glennzw can you be more specific on your environment? You should not have your dev server public on the open web accessible by bots. :) In such a situation, like a demo host for clients, it's always better to use at least a real application server like gunicorn which has a really small footprint. |
FWIW, I've haven't been keeping up with releases much and this (very annoying) problem started happening for me sometime between May - August 2016 best I can tell. I added this to my setup.py For me the duplication case was simple enough - load a page, but don't let it finish loading. That is, just trigger any broken pipe error - and the webserver will crash and fail to service any other requests. IMHO, webservers can't fall over when a client closes the connection prematurely - even development ones. |
Could it be that you all have |
@untitaker in that case, pallets/flask#1674 pallets/flask#1679 pallets/flask#1928 possibly related? |
I don't know, I would like one of the reporters to confirm. On 26 August 2016 17:05:25 CEST, David Lord notifications@github.com wrote:
Sent from my Android device with K-9 Mail. Please excuse my brevity. |
I think Werkzeug should handle broken pipe and connection reset errors. These are really not an indication of an error, the client just went away. Seems in this case a special exception should be raised, one that is recognized by the catch-all way above as one to ignore, even if error passthrough is set. Here is how gunicorn does it: https://github.com/benoitc/gunicorn/blob/39f62ac66beaf83ceccefbfabd5e3af7735d2aff/gunicorn/workers/sync.py#L151-L154 |
That's what it is supposed to do. I'm trying to figure out how to reproduce I suspect that this is not a bug in Werkzeug, and that the user's browser On Fri, Aug 26, 2016 at 11:54:16AM -0700, Miguel Grinberg wrote:
|
Oh, and broken pipe errors are displayed, yes, but they should never hang up the server like described. See previous comment about the possible reason. |
I tested again the latest bits and I'm still seeing the same behavior in my environment. But since you seemed to be having trouble reproducing I tried to pull out why I'm special.
It work as expected with |
It appears a part of your response is lost. On Fri, Aug 26, 2016 at 12:29:39PM -0700, clayg wrote:
|
yes, in slack triple-ticks is how you block quote and I edited my post immediately after submitting it to finish up - and I'm only responding because it appears you're replying from email and I'm not sure github will send you another notice after my edit. |
I cannot reproduce with the sleep test by @clayg above. I actually don't get the broken pipe error at all. I close the browser before that request gets to return a response, but Werkzeug carries out the request until the end anyway, it prints the 200 log line to the console and does not show any error. I also tried the same trick using my flask video streaming example which uses a streaming response to provide video frames in a never ending stream, and even for that one, I can close the browser and the request ends without any errors. This one is weird, because I am certain in the past this application would trigger a broken pipe error to the console before ending the request. |
Actually, I spoke too soon. I can reproduce every time with my video streaming app when using Python 2.7. I cannot reproduce on 3.5. All the stack traces above are for 2.7, so keep that in mind if you are testing with Python 3. One more interesting data point. If running with the reloader, when the child process exits, the master process running the reloader starts another one, so there is no interruption. But if you are running the server without the reloader, the broken pipe error brings you back to the console. Edit: disregard the reloader starting another process part, that does not seem to be happening and instead I was likely seeing the effect of changing the passthrough error setting. |
Okay, here is the analysis of what I think is happening:
The socket code on Python 3 is completely different, and in particular, it does not appear to have any flush calls without try/excepts around them. The EPIPE error does not even bubble up to Werkzeug when using Python 3. |
Do we even have passthrough_errors set to true? In Werkzeug it is false by default. On 27 August 2016 02:10:13 CEST, Miguel Grinberg notifications@github.com wrote:
Sent from my Android device with K-9 Mail. Please excuse my brevity. |
Oh, ahem: pallets/flask#1679
|
I see no option other than reverting that PR actually. |
Nevermind, I found another way. Two PRs:
Since both are behavior changes in the broad sense, I'd rather not backport them. |
I think pallets/flask#1996 is an acceptable solution. The important thing is that it fixes the common case where you do not want exceptions propagated. If you do want to propagate, then you are debugging, and in that case getting the socket.error propagated when it shouldn't is not a huge deal. The #998 fix is not great though. The application might raise these exceptions legitimately from something it is doing with sockets in its own handlers, and those would be silenced as well. The ideal solution would be that these are caught at the place they occur and then re-raised as some custom exception class that |
The behavior you describe only happens with PASSTHROUGH_ERRORS enabled though. Otherwise the exception is caught from within Flask. I guess this cosmetic improvement is not worth it though. On 27 August 2016 18:29:30 CEST, Miguel Grinberg notifications@github.com wrote:
Sent from my Android device with K-9 Mail. Please excuse my brevity. |
Fixed in master. |
Yes, I omitted that detail. But this change would even affect Python 3, where none of this is an issue. On Py3, with passthrough errors enabled, a legitimate socket.error raised by the application would be silenced. |
master seems to wfm, looking forward to the next release, thanks! |
Hi , I use a Werkzeug dev server running behind NGINX, I am facing the same issue could anyone help me with this , Please help |
Ragav use another application server instead of werkzeug's built in dev 2016-09-15 8:07 GMT+02:00 Ragav notifications@github.com:
|
Guided by the recent fix commit, I've been able to fix this problem by calling app.run with passthrough_errors=False. YMMV |
The bug that caused the crash was fixed in Version 0.12, Released on December 21st 2016.
|
Version 0.12 has only been released last week.
…On Mon, Mar 20, 2017 at 09:05:00AM -0700, Alan Rotman wrote:
The bug that caused the crash **was fixed in Version 0.12**, Released on December 21st 2016.
- Revert a behavior change that made the dev server crash instead of returning a Internal Server Error (pull request #2006).
--
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
#954 (comment)
|
I just saw ReleaseNotes today, and have been waiting for this fix for a long time. Look at: http://flask.pocoo.org/docs/0.12/changelog/ https://pypi.python.org/pypi/Flask/0.12 |
Ah, yes, you mean Flask. Sure.
…On Mon, Mar 20, 2017 at 09:22:15AM -0700, Alan Rotman wrote:
I just saw ReleaseNotes today, and have been waiting for this fix for a long time.
Look at: http://flask.pocoo.org/docs/0.12/changelog/
Version 0.12
Released on December 21st 2016, codename Punsch.
https://pypi.python.org/pypi/Flask/0.12
File Type Py Version Uploaded on Size
Flask-0.12-py2.py3-none-any.whl (md5) Python Wheel 2.7 2016-12-21 80KB
Flask-0.12.tar.gz (md5) Source 2016-12-21 519KB
--
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
#954 (comment)
|
Just a note for anyone who runs across this issue when running flask 0.12.2 on werkzeug in threaded=True mode:
And then the server continues to run as normal. I still have to wait a few hours to see if the memory leak shows up again, but I'm hopeful that it won't. Just in case that's helpful for anyone. |
I saw this error too in an Ubuntu Docker container on Kubernetes on Ubuntu VM:
I created a brand new Ubuntu xenial VM and ran the same code in Ubuntu Docker container on Kubernetes, and this error was not seen and Python Flask worked as expected. I think it was issue with my host (Ubuntu VM). |
@vhosakot Can you let me know how you set up your app configuration? I encountered a similar issue in the same environment as yours. |
In a route function, I used another function which was meant for routing.
Error: |
Fix if the connection is closed early by remote client. Long history pallets/werkzeug#954 Close issues #97, #179 also mentioned in #168
I have a Werkzeug server running behind NGINX. When a client disconnects while waiting for the Werkzeug server to respond, NGINX closes the pipe to Werkzeug. When the python program writes the response to Werkzeug, the following exception occurs and Werkzeug crashes:
Is there some configuration option I'm missing to keep it from crashing? Normally all exceptions are caught and a 500 error returned, with the server remaining alive.
The text was updated successfully, but these errors were encountered: