[subinterpreters] PEP 554 implementation: add interpreters module #76785
Comments
|
In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module. In some ways this is a precursor for issue bpo-30439, which will add a proper public stdlib module in 3.8. The module I'm adding draws from the ideas in PEP-554 (particularly for channels). Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8. I expect to have some follow-on patches to facilitate simpler use in tests. This patch is big enough already. :) |
ericsnowcurrently
added
3.7 (EOL)
|
@ned, it may be a little tight to land this given the time left before beta 1. However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API). So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze. (I'm still hoping to get this in before the cutoff.) What do you think? |
|
FYI, there are a few things I need to clean up in the PR. However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :) |
|
@eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time. If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers. |
|
Sounds good, Ned. Thanks for taking a look. I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline. I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens. |
|
I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module. I've also added a PR for get the module building under Windows. I'd like to get that resolved ASAP. |
|
Eric, looks like some buildbots are unhappy, for instance: |
|
Yeah, I'm looking into it. Also, I noticed some refleaks that I'll be sorting out. |
|
On 4 of the buildbots: ====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3) |
|
On the PPC64 AIX 3.x buildbot: ====================================================================== Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test__xxsubinterpreters.py", line 784, in test_repr
self.assertEqual(repr(cid), 'ChannelID(10)')
AssertionError: 'ChannelID(0)' != 'ChannelID(10)'
- ChannelID(0)
+ ChannelID(10)
? + |
|
I just put up a PR that should fix the 4 buildbots. |
|
The buildbots should be happier now. I'll keep an eye on them. |
|
A couple defects reported by coverity: ** CID 1428758: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 45 in _coerce_id() 1217 } |
|
I've landed a PR that fixes all the memory leaks in the module. It also fixes the 2 defects reported by coverity. The only thing left here is to get the module building under Windows. |
|
FYI, out of 2389 source lines in the C extension, 1563 are the channel-related code. That means the non-channel code is 826 lines (about a third). That non-channel code does not depend on the channel code at all and I considered splitting the source out, but figured there wasn't enough benefit. However, I might revisit the matter later when I circle back to PEP-554. :) |
|
Eric, it looks like your recent commit introduced a refleak. We need to fix it before beta2. ~/d/p/cpython (master $) » ./python.exe -m test -R3:3 test_multiprocessing_fork 1 test failed: And just before it: ~/d/p/cpython ((bd09335…) $) » ./python.exe -m test -R3:3 test_multiprocessing_fork Total duration: 9 min 12 sec |
…nalization (pythongh-121805) See 6b98b27 for an explanation of the problem and solution. Here I've applied the solution to channels. (cherry picked from commit 8b209fd) Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
…lization (pythongh-116431) Any cross-interpreter mechanism for passing objects between interpreters must be very careful to respect isolation, even when the object is effectively immutable (e.g. int, str). Here this especially relates to when an interpreter sends one of its objects, and then is destroyed while the inter-interpreter machinery (e.g. queue) still holds a reference to the object. When I added interpreters.Queue, I dealt with that case (using an atexit hook) by silently removing all items from the queue that were added by the finalizing interpreter. Later, while working on concurrent.futures.InterpreterPoolExecutor (pythongh-116430), I noticed it was somewhat surprising when items were silently removed from the queue when the originating interpreter was destroyed. (See my comment on that PR.) It took me a little while to realize what was going on. I expect that users, which much less context than I have, would experience the same pain. My approach, here, to improving the situation is to give users three options: 1. return a singleton (interpreters.queues.UNBOUND) from Queue.get() in place of each removed item 2. raise an exception (interpreters.queues.ItemInterpreterDestroyed) from Queue.get() in place of each removed item 3. existing behavior: silently remove each item (i.e. Queue.get() skips each one) The default will now be (1), but users can still explicitly opt in any of them, including to the silent removal behavior. The behavior for each item may be set with the corresponding Queue.put() call. and a queue-wide default may be set when the queue is created. (This is the same as I did for "synconly".)
…nalization (pythongh-121805) See 6b98b27 for an explanation of the problem and solution. Here I've applied the solution to channels.
…ta (pythongh-109556) This fixes some crashes in the _xxinterpchannels module, due to a race between interpreters.
…hongh-110236) * add RecvChannel.close() and SendChannel.close() * make RecvChannel and SendChannel shareable * expose ChannelEmptyError and ChannelNotEmptyError
…110248) This is a temporary fix. The full fix may involve serializing the traceback in some form.
…ythongh-110248)" (pythongh-110314) This reverts commit 6bc889a. That commit is causing some buildbot failures.
…110322) This is a temporary solution. The full fix may involve serializing the traceback in some form. (FYI, I merged this yesterday and the reverted it due to buildbot failures. See pythongh-110248.)
…thongh-110251) This specifically refers to `test.support.interpreters.Interpreter.run()`.
(This is still a test module.)
…h-110607) This restores their shareability, which was disabled by pythongh-110318 due to ref leaks.
…ongh-111502) This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c. This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h). @vstinner did this previously and I undid it. Now I'm re-doing it. :/
This moves several general internal APIs out of _xxsubinterpretersmodule.c and into the new Python/crossinterp.c (and the corresponding internal headers). Specifically: * _Py_excinfo, etc.: the initial implementation for non-object exception snapshots (in pycore_pyerrors.h and Python/errors.c) * _PyXI_exception_info, etc.: helpers for passing an exception beween interpreters (wraps _Py_excinfo) * _PyXI_namespace, etc.: helpers for copying a dict of attrs between interpreters * _PyXI_Enter(), _PyXI_Exit(): functions that abstract out the transitions between one interpreter and a second that will do some work temporarily Again, these were all abstracted out of _xxsubinterpretersmodule.c as generalizations. I plan on proposing these as public API at some point.
There were a few corner cases I didn't handle properly in pythongh-111530, which I've noticed while working on a follow-up PR. This fixes those cases.
…ythongh-111715) I added _Py_excinfo to the internal API (and added its functions in Python/errors.c) in pythongh-111530 (9322ce9). Since then I've had a nagging sense that I should have added the type and functions in its own PR. While I do plan on using _Py_excinfo outside crossinterp.c very soon (see pythongh-111572/pythongh-111573), I'd still feel more comfortable if the _Py_excinfo stuff went in as its own PR. Hence, here we are. (FWIW, I may combine that with pythongh-111572, which I may, in turn, combine with pythongh-111573. We'll see.)
…thongh-112323) The new function corresponds to the existing (public) PyType_GetName() and PyType_GetQualName().
This involves a number of changes for PEP 734.
…ongh-113014) pythongh-112982 broke test_threading on one of the s390 buildbots (Fedora Clang Installed). Apparently ImportError is raised (rather than ModuleNotFoundError) for the name part of "from" imports. This fixes that by catching ImportError in test_threading.py.
…3012) This brings the module (along with the associated extension modules) mostly in sync with PEP 734. There are only a few small things to wrap up.
This is one of the last pieces to get test.support.interpreters in sync with PEP 734.
…ions (pythongh-113034) When an exception is uncaught in Interpreter.exec_sync(), it helps to show that exception's error display if uncaught in the calling interpreter. We do so here by generating a TracebackException in the subinterpreter and passing it between interpreters using pickle.
…terpreter Exceptions (pythongh-113036) We need the TracebackException of uncaught exceptions for a single purpose: the error display. Thus we only need to pass the formatted error display between interpreters. Passing a pickled TracebackException is overkill.
In pythongh-112982 I made some changes to .github/CODEOWNERS. Later, @ezio-melotti pointed out that some of those changes were unnecessary.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: