Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Python Critical Sections from PEP 703 #111569

Closed
Tracked by #108219
colesbury opened this issue Oct 31, 2023 · 1 comment
Closed
Tracked by #108219

Implement Python Critical Sections from PEP 703 #111569

colesbury opened this issue Oct 31, 2023 · 1 comment
Assignees
@colesbury
Labels
3.13 bugs and security fixes OS-wasi topic-free-threading type-feature A feature request or enhancement

Comments

@colesbury
Copy link
Contributor

colesbury commented Oct 31, 2023 •
edited by bedevere-app bot
Loading

Feature or enhancement

PEP 703 introduces the concept of "Python Critical Sections", which are an abstraction to help replace the GIL with finer grained locking. The key ideas is to replace the GIL with per-object locks and implicitly release these locks in the same places where the GIL would have been released. The mechanism is:

  1. Keep track of locked mutexes in a per-thread stack
  2. When the thread "detaches" from the interpreter (i.e., _PyThreadState_Detach()), unlock all of the thread's locked mutexes
  3. When the thread re-"attaches" to the interpreter, re-lock the top-most mutex or mutexes.

The "public" 1 API consists of four macros:

Py_BEGIN_CRITICAL_SECTION(object);
Py_END_CRITICAL_SECTION;
Py_BEGIN_CRITICAL_SECTION2(object1, object2);
Py_END_CRITICAL_SECTION2;

These will be no-ops in the default build of CPython.

Note that if you need to operate on two objects at once, then you must use the Py_BEGIN_CRITICAL_SECTION2 macro. Nesting two calls to Py_BEGIN_CRITICAL_SECTION does not guarantee that both objects are locked because the inner calls may suspend the outer critical section.

Linked PRs

Footnotes

  1. At least at the start, even these "public" macros will still be internal-only (i.e., in Include/internal). I expect that we will eventually want to make them public so that C-API extensions can make use of them. ↩
@colesbury colesbury added type-feature A feature request or enhancement 3.13 bugs and security fixes topic-free-threading labels Oct 31, 2023
@colesbury colesbury mentioned this issue Oct 31, 2023
Open
@colesbury colesbury self-assigned this Oct 31, 2023
colesbury added a commit to colesbury/cpython that referenced this issue Oct 31, 2023
@colesbury
pythongh-111569: Implement Python critical section API
6bf34f5 
Critical sections are helpers to replace the global interpreter lock
with finer grained locking. They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves. Critical
sections are implicitly ended whenever the GIL would be released. They
are resumed when the GIL would be acquired. Nested critical sections
behave as if the sections were interleaved.
@bedevere-app bedevere-app bot mentioned this issue Oct 31, 2023
Merged
@colesbury colesbury mentioned this issue Nov 2, 2023
Closed
ericsnowcurrently pushed a commit that referenced this issue Nov 8, 2023
@colesbury
gh-111569: Implement Python critical section API (gh-111571)
31c90d5 
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
@brettcannon
Copy link
Member

The latest PR related to this change broke the WebAssembly buildbots by causing them to time out: https://buildbot.python.org/all/#/builders/1046/builds/3456/steps/11/logs/stdio .

colesbury added a commit to colesbury/cpython that referenced this issue Nov 9, 2023
@colesbury
pythongh-111569: Fix critical sections test on WebAssembly
aef9180 
This adds a macro `Py_CAN_START_THREADS` that corresponds to the Python
function `test.support.threading_helper.can_start_thread()`. WASI and
some Emscripten builds do not have a working pthread implementation.

This macro is used to guard the critical sections C API tests that
require a working threads implementation.
@bedevere-app bedevere-app bot mentioned this issue Nov 9, 2023
Merged
@colesbury colesbury mentioned this issue Nov 9, 2023
Closed
brettcannon pushed a commit that referenced this issue Nov 9, 2023
@colesbury
gh-111569: Fix critical sections test on WebAssembly (GH-111897)
289af86 
This adds a macro `Py_CAN_START_THREADS` that corresponds to the Python
function `test.support.threading_helper.can_start_thread()`. WASI and
some Emscripten builds do not have a working pthread implementation.

This macro is used to guard the critical sections C API tests that
require a working threads implementation.
@colesbury colesbury mentioned this issue Jan 18, 2024
Open
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@colesbury @aisk
pythongh-111569: Implement Python critical section API (pythongh-111571)
7907203 
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@colesbury @aisk
pythongh-111569: Fix critical sections test on WebAssembly (pythonGH-…
41586cd 
…111897)

This adds a macro `Py_CAN_START_THREADS` that corresponds to the Python
function `test.support.threading_helper.can_start_thread()`. WASI and
some Emscripten builds do not have a working pthread implementation.

This macro is used to guard the critical sections C API tests that
require a working threads implementation.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@colesbury @Glyphack
pythongh-111569: Implement Python critical section API (pythongh-111571)
3b0497b 
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@colesbury @Glyphack
pythongh-111569: Fix critical sections test on WebAssembly (pythonGH-…
97032d5 
…111897)

This adds a macro `Py_CAN_START_THREADS` that corresponds to the Python
function `test.support.threading_helper.can_start_thread()`. WASI and
some Emscripten builds do not have a working pthread implementation.

This macro is used to guard the critical sections C API tests that
require a working threads implementation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@colesbury colesbury

Labels
3.13 bugs and security fixes OS-wasi topic-free-threading type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brettcannon @colesbury