-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-37380: subprocess: skip _cleanup and __del__ on win #14360
Conversation
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You can check yourself to see if the CLA has been received. Thanks again for your contribution, we look forward to reviewing it! |
Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
Outdated
Show resolved
Hide resolved
636dfcf
to
f59cad6
Compare
Thanks for the review @eryksun ! Addressed your comments in the patch. |
Lib/subprocess.py
Outdated
if _mswindows: | ||
# We don't need to do anything in Windows. See `_cleanup` comment | ||
# in `Popen.__init__` for more info. | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's wrong: you must also wait for process completion on Windows.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @vstinner ! Thanks for the review! On Windows we just need to close the handle when we no longer need it, so that the kernel can free it. CloseHandle
is called as a __del__
in class Handle
defined above, so there are no additional actions required here, and so we can just skip the whole __del__
logic here. From [1]
When you no longer need these handles, close them by using the CloseHandle function.
[1] https://docs.microsoft.com/en-us/windows/desktop/ProcThread/creating-processes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vstinner If we're in __del__
, presumably that means that nobody has a reference to the Popen
object, right? In that case, we can let it be cleaned up. Closing the handle does not terminate the process - it'll just keep running and we can't get back to it through that particular handle (but we could use OpenProcess
with the PID to get a new handle later).
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
eb04016
to
80d8568
Compare
Lib/subprocess.py
Outdated
if _mswindows: | ||
# We don't need to do anything in Windows. See `_cleanup` comment | ||
# in `Popen.__init__` for more info. | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vstinner If we're in __del__
, presumably that means that nobody has a reference to the Popen
object, right? In that case, we can let it be cleaned up. Closing the handle does not terminate the process - it'll just keep running and we can't get back to it through that particular handle (but we could use OpenProcess
with the PID to get a new handle later).
Steve:
My worry is that a badly written program may "leak" programs which continue to run in the background which can cause different issues. For example, you may not be able to remove the directory used by these background tasks since they have a handle open on these directories. It can cause various different issues. The idea of adding a detatch() method to Popen was proposed once but didn't convince enough people to be implemented: https://bugs.python.org/issue27068 The purpose of the ResourceWarning is to help the developer to write more reliable softwares. It's just a matter of a handle which remains open longer than expected. I'm talking about this warning:
I would like to also get this warning on Windows. |
Guys, big thanks for the reviews! I've adjusted my patch, so that it only re-defines |
I don't see where this PR explicitly closes the handle: https://bugs.python.org/issue37380#msg346568 |
As to
The process exited. It is not running. The warning is wrong. |
In a related issue to waiting on the child process, if someone wants to open an easy issue for this, we should not be creating |
The purpose is to promote to handle explicitly resources, rather than relying on Python garbage collector to magically handle them: the garbage collection behaves differently on PyPy, and may not collect objects "immediately" depending if they are part of a reference cycle or not. subprocess.Popen should "release" its resources when the process exit: close pipes, "close" the child processes, etc. I suggest to use "with popen: ..." since exit() closes pipes and waits until the child process completes. |
In the case of Windows, I would expect that the handle can be closed as soon as the process completes. But it seems to be the case now: https://bugs.python.org/issue37380#msg346568 |
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380#msg346333 [2] https://bugs.python.org/issue36067#msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>
Adjusted the |
# implicitly when the `Popen` instance is finalized (see `Handle.__del__`, | ||
# which is calling `CloseHandle` as requested in [1]), so there is nothing | ||
# for `_cleanup` to do. | ||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
FYI, if bpo37410 is resolved, we'll also be explicitly closing _handle
when returncode
is set. This would be a significant change. Currently we can assume _handle
is valid, but with the proposed change we'll have to acquire a lock and check whether self.returncode is None
before code that sets it in _internal_poll
, _wait
, and terminate
.
GH-15706 is a backport of this pull request to the 3.7 branch. |
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380GH-msg346333 [2] https://bugs.python.org/issue36067GH-msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai> (cherry picked from commit 042821a) Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
GH-15707 is a backport of this pull request to the 3.8 branch. |
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380GH-msg346333 [2] https://bugs.python.org/issue36067GH-msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai> (cherry picked from commit 042821a) Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380GH-msg346333 [2] https://bugs.python.org/issue36067GH-msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai> (cherry picked from commit 042821a) Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380GH-msg346333 [2] https://bugs.python.org/issue36067GH-msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai> (cherry picked from commit 042821a) Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380#msg346333 [2] https://bugs.python.org/issue36067#msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>
As noted by @eryksun in [1] and [2], using _cleanup and _active(in __del__) is not necessary on Windows, since: > Unlike Unix, a process in Windows doesn't have to be waited on by > its parent to avoid a zombie. Keeping the handle open will actually > create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. This patch simply defines `subprocess._active` as `None`, for which we already have the proper logic in place in `subprocess.Popen.__del__`, that prevents it from trying to append the process to the `_active`. This patch also defines `subprocess._cleanup` as a noop for Windows. [1] https://bugs.python.org/issue37380#msg346333 [2] https://bugs.python.org/issue36067#msg336262 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>
To better reflect python/cpython#14360
To better reflect python/cpython#14360
As noted by @eryksun in [1] and [2], using _cleanup and _active(in
del) is not necessary on Windows, since:
This patch simply defines
subprocess._active
asNone
, for which we alreadyhave the proper logic in place in
subprocess.Popen.__del__
, that prevents itfrom trying to append the process to the
_active
. This patch also definessubprocess._cleanup
as a noop for Windows.[1] https://bugs.python.org/issue37380#msg346333
[2] https://bugs.python.org/issue36067#msg336262
Signed-off-by: Ruslan Kuprieiev ruslan@iterative.ai
https://bugs.python.org/issue37380