Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make multiprocessing SemLock thread-safe in the free-threaded build #117435

Closed
Tracked by #108219
colesbury opened this issue Apr 1, 2024 · 0 comments
Closed
Tracked by #108219

Make multiprocessing SemLock thread-safe in the free-threaded build #117435

colesbury opened this issue Apr 1, 2024 · 0 comments
Labels
topic-free-threading type-feature A feature request or enhancement

Comments

@colesbury
Copy link
Contributor

colesbury commented Apr 1, 2024

Feature or enhancement

The multiprocessing SemLock code maintains an internal count field. When operating as a recursive mutex, count is the number of times the thread has acquired the mutex (i.e., 0, 1, .. N). When operating as a semaphore, count is often 0 or 1, but can be negative if the SemLock is initialized with maxvalue > 1

The modification of count is not thread-safe without the GIL within the process. Note that the count field is not shared across processes, unlike the underlying sem_t or HANDLE.

In particular, the modification of count after the semaphore is released is not thread-safe without the GIL:

if (sem_post(self->handle) < 0)
return PyErr_SetFromErrno(PyExc_OSError);
--self->count;
Py_RETURN_NONE;

I think this is the source of deadlocks when running test_multiprocessing_forkserver.test_processes with the GIL disabled.

Linked PRs

@colesbury colesbury added type-feature A feature request or enhancement topic-free-threading labels Apr 1, 2024
@colesbury colesbury changed the title Make multiprocessing semaphore thread-safe in the free-threaded build Make multiprocessing SemLock thread-safe in the free-threaded build Apr 1, 2024
colesbury added a commit to colesbury/cpython that referenced this issue Apr 1, 2024
Use critical sections to make acquire, release, and _count thread-safe
without the GIL.
colesbury added a commit to colesbury/cpython that referenced this issue Apr 1, 2024
colesbury added a commit to colesbury/cpython that referenced this issue Apr 2, 2024
colesbury added a commit that referenced this issue Apr 4, 2024
Use critical sections to make acquire, release, and _count thread-safe
without the GIL.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
…ython#117436)

Use critical sections to make acquire, release, and _count thread-safe
without the GIL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-free-threading type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

1 participant