-
-
Notifications
You must be signed in to change notification settings - Fork 30.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters #103084
Merged
ericsnowcurrently
merged 14 commits into
python:main
from
ericsnowcurrently:extensions-dict-owned-by-main-interpreter-2
Mar 29, 2023
Merged
gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters #103084
ericsnowcurrently
merged 14 commits into
python:main
from
ericsnowcurrently:extensions-dict-owned-by-main-interpreter-2
Mar 29, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericsnowcurrently
requested review from
brettcannon,
encukou,
ncoghlan and
warsaw
as code owners
March 28, 2023 14:56
ericsnowcurrently
added
the
🔨 test-with-refleak-buildbots
Test PR w/ refleak buildbots; report in status section
label
Mar 29, 2023
🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit e9c37fe 🤖 If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again. |
bedevere-bot
removed
the
🔨 test-with-refleak-buildbots
Test PR w/ refleak buildbots; report in status section
label
Mar 29, 2023
ericsnowcurrently
deleted the
extensions-dict-owned-by-main-interpreter-2
branch
March 29, 2023 23:15
ericsnowcurrently
added a commit
that referenced
this pull request
Mar 31, 2023
warsaw
pushed a commit
to warsaw/cpython
that referenced
this pull request
Apr 11, 2023
…Interpreters (pythongh-103084) Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily. There are two restrictions that are critical for objects that break interpreter isolation. The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL. The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, see pythongh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes. The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following: * add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread * add _PyRuntime.imports.extensions.main_tstate` * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c) * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c) * make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter) * add a placeholder using for a granular global lock Note that the cache is only used for legacy extension modules and not for multi-phase init modules. python#100227
warsaw
pushed a commit
to warsaw/cpython
that referenced
this pull request
Apr 11, 2023
Decref the key in the right interpreter in _extensions_cache_set(). This is a follow-up to pythongh-103084. I found the bug while working on pythongh-101660.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily.
There are two restrictions that are critical for objects that break interpreter isolation.
The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.
The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, see gh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes.
The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following:
_PyRuntime.imports.extensions.main_tstate
_PyThreadState_InitDetached()
and_PyThreadState_ClearDetached()
(pystate.c)_PyThreadState_BindDetached()
and_PyThreadState_UnbindDetached()
(pystate.c)_PyRuntime.imports.extensions.dict
) and its items are all owned by the main interpreter)Note that the cache is only used for legacy extension modules and not for multi-phase init modules.
(This PR was derived from gh-102925, which I recently reverted. Also, there was a similar, more complex PR that I've closed: gh-102938.)