Close listening socket immediately after stop signal received #922
Comments
|
Short upstream timeouts reduce the impact, but is not ideal. As @benoitc has pointed out, we cannot simply close the socket on any graceful shutdown because the worker might be shutting down after a new master is spawned with USR2. |
|
Normally, spawning (forking) new master should be before stopping current master. After fork, it's totally safe to close listening socket. The socket will remain open in the forked child. (sample code) |
|
True enough. I'll try to dig up the original conversation, in case there's Thanks, tahajahangir Do you want to try writing a patch? I'm happy to review. Otherwise, I can
|
|
Unfortunately, I'm not familiar with gunicorn structure, additionally we moved to uwsgi in production, and I don't have enough free time to dig into gunicorn. |
|
So the socket will indeed remains in the forked child but we should make sure to kill the socket in the parent only on final exit (ie when the the master is finally asked to exit) during USR2. I will work on such patch asap. |
benoitc
added
Improvement
Feature/Core
working on it :)
and removed
working on it :)
labels
|
Was this ever fixed? |
|
I din't tested it yet. thanks for the head-up it's on the todo for today On Mon, May 18, 2015 at 9:55 AM Aris Pikeas notifications@github.com
|
|
+1 on getting this in |
|
Re-reading this issue, I think we are missing some details, was the server closed using |
|
We're not closing the socket immediately which I think means that a On Thu, Dec 31, 2015, 02:16 Benoit Chesneau notifications@github.com
|
Close all the listeners when the arbiter shuts down. By doing so, workers can close the socket at the beginning of a graceful shut down thereby informing the operating system that the socket can be cleaned up. With this change, graceful exits with such workers will refuse new connections while draining, allowing load balancers to respond more quickly and avoiding leaving connections dangling in the listen backlog, unaccepted. Ref #922
Close all the listeners when the arbiter shuts down. By doing so, workers can close the socket at the beginning of a graceful shut down thereby informing the operating system that the socket can be cleaned up. With this change, graceful exits with such workers will refuse new connections while draining, allowing load balancers to respond more quickly and avoiding leaving connections dangling in the listen backlog, unaccepted. Ref #922
|
The run loop has to change slightly to support graceful shutdown. There is no way to interrupt a call to `futures.wait` so instead the pattern, used by the async workers, is to sleep for only one second at the most. The poll is extended to a one second timeout to match. Since threads are preemptively scheduled, it's possible that the listener is closed when the request is actually handled. For this reason it is necessary to slightly refactor the TConn class to store the listening socket name. The name is checked once at the start of the worker run loop. Ref #922
|
I believe that this can be marked complete as soon as the two PRs I have open for eventlet and the threaded worker are merged. |
|
@tilgovi 👍 |
|
Closing as all my open PRs have been merged. Opened #1236 to track future work arising from this discussion and related PRs. |
Close all the listeners when the arbiter shuts down. By doing so, workers can close the socket at the beginning of a graceful shut down thereby informing the operating system that the socket can be cleaned up. With this change, graceful exits with such workers will refuse new connections while draining, allowing load balancers to respond more quickly and avoiding leaving connections dangling in the listen backlog, unaccepted. Ref benoitc#922
The run loop has to change slightly to support graceful shutdown. There is no way to interrupt a call to `futures.wait` so instead the pattern, used by the async workers, is to sleep for only one second at the most. The poll is extended to a one second timeout to match. Since threads are preemptively scheduled, it's possible that the listener is closed when the request is actually handled. For this reason it is necessary to slightly refactor the TConn class to store the listening socket name. The name is checked once at the start of the worker run loop. Ref benoitc#922
When stopping gunicorn, server should wait for current requests to complete, but this may take several seconds (even 30 seconds, with the default timeout). Clients connecting in this interval, will wait forever, and will receive connection-reset error after server successfully stopped.
When using two or more gunicorn servers behind a reverse proxy (like nginx), this behavior leads to unreasonable raise in total upstream time (normally, real clients won't wait for 30 seconds, and will close tab, or hit refresh).
This also reduces impact of #901 (locked connections). Also related to #908
The text was updated successfully, but these errors were encountered: