Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deadlocked child process after forking on pystate.c's HEAD_LOCK #112275

Closed
ChuBoning opened this issue Nov 20, 2023 · 5 comments
Closed

deadlocked child process after forking on pystate.c's HEAD_LOCK #112275

ChuBoning opened this issue Nov 20, 2023 · 5 comments
Labels
3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) pending The issue will be closed if no feedback is provided topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@ChuBoning
Copy link

ChuBoning commented Nov 20, 2023 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

A forked process (via os.fork) can inherit a locked runtime->interpreters.mutex from its parent process and will deadlock in this stack:
Child Process (deadlocked):
#0 0x0000ffffa9b2b268 in do_futex_wait.constprop () from /lib/aarch64-linux-gnu/libpthread.so.0
#1 0x0000ffffa9b2b39c in __new_sem_wait_slow.constprop.0 () from /lib/aarch64-linux-gnu/libpthread.so.0
#2 0x0000ffffa9e96eb8 in PyThread_acquire_lock_timed () from /usr/local/lib/libpython3.8.so.1.0
#3 0x0000ffffa9e865a8 in _PyThreadState_DeleteExcept () from /usr/local/lib/libpython3.8.so.1.0
#4 0x0000ffffa9eb94ac in PyOS_AfterFork_Child () from /usr/local/lib/libpython3.8.so.1.0

Similar to #74580 and it's solved by f82c951 on python3.6 and 3.7

However,the problem appeares again on python3.8: HEAD_LOCK is called from PyOS_AfterFork_Child->_PyEval_ReInitThreads->_PyThreadState_DeleteExcept before PyOS_AfterFork_Child->_PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex.

I notice the problem will be resolved on python3.12 from the source code,but is it still existed on python3.8 to 3.11?

CPython versions tested on:

3.8

Operating systems tested on:

Linux

Linked PRs
@ChuBoning ChuBoning added the type-bug An unexpected behavior, bug, or error label Nov 20, 2023
@arhadthedev arhadthedev added interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-multiprocessing 3.11 only security fixes labels Nov 20, 2023
@itamaro
Copy link
Contributor

itamaro commented Nov 20, 2023

Python 3.8 is receiving only security fixes currently (so is 3.9 and 3.10).

Could you check whether this issue reproduces with 3.11 and/or 3.12 and update the issue accordingly?

@itamaro itamaro added the pending The issue will be closed if no feedback is provided label Nov 20, 2023
@ChuBoning
Copy link
Author

Python 3.8 is receiving only security fixes currently (so is 3.9 and 3.10).

Could you check whether this issue reproduces with 3.11 and/or 3.12 and update the issue accordingly?

I haven't tried it on 3.11 yet.But I think this issue will reproduce on 3.11. And there is no such issue on 3.12 since _PyRuntimeState_ReInitThreads calls _PyThread_at_fork_reinit to reinit the lock at the beginning.

@bedevere-app bedevere-app bot mentioned this issue Nov 23, 2023
Merged
@ChuBoning
Copy link
Author

Python 3.8 is receiving only security fixes currently (so is 3.9 and 3.10).

Could you check whether this issue reproduces with 3.11 and/or 3.12 and update the issue accordingly?

Could you please take a look at this #112336 ? I fix it just as python3.12 does.

ambv added a commit that referenced this issue Sep 4, 2024
@ChuBoning @ambv
[3.11]GH-112275: Fix HEAD_LOCK deadlock in child process after fork (#…
522799a 
…112336)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 4, 2024
@ChuBoning @ambv @miss-islington
[3.11]pythonGH-112275: Fix HEAD_LOCK deadlock in child process after …
bbeaba7 
…fork (pythonGH-112336)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)

Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
@bedevere-app bedevere-app bot mentioned this issue Sep 4, 2024
Merged
ambv added a commit to ambv/cpython that referenced this issue Sep 4, 2024
@ChuBoning @ambv
[3.9] [3.11]pythonGH-112275: Fix HEAD_LOCK deadlock in child process …
48c606a 
…after fork (pythonGH-112336)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)

Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
ambv added a commit that referenced this issue Sep 4, 2024
@miss-islington @ChuBoning @ambv
[3.10] gh-112275: Fix HEAD_LOCK deadlock in child process after fork (G…
d3f39ce 
…H-112336) (#123687)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)

Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
@bedevere-app bedevere-app bot mentioned this issue Sep 4, 2024
Merged
ambv added a commit that referenced this issue Sep 4, 2024
@ambv @ChuBoning
[3.9] gh-112275: Fix HEAD_LOCK deadlock in child process after fork (G…
0152431 
…H-112336) (#123688)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)

Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 4, 2024
@ambv @ChuBoning @miss-islington
[3.9] pythongh-112275: Fix HEAD_LOCK deadlock in child process after …
a8e5483 
…fork (pythonGH-112336) (pythonGH-123688)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)

(cherry picked from commit 0152431)

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
@bedevere-app bedevere-app bot mentioned this issue Sep 4, 2024
Merged
@ChuBoning
Copy link
Author

Thanks a lot!

ambv added a commit that referenced this issue Sep 5, 2024
@miss-islington @ambv @ChuBoning
[3.8] gh-112275: Fix HEAD_LOCK deadlock in child process after fork (G…
e742f99 
…H-112336) (GH-123688) (#123713)

HEAD_LOCK is called from _PyEval_ReInitThreads->_PyThreadState_DeleteExcept before _PyRuntimeState_ReInitThreads reinit runtime->interpreters.mutex which might be locked before fork.

(cherry picked from commit 522799a)
(cherry picked from commit 0152431)

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: ChuBoning <102216855+ChuBoning@users.noreply.github.com>
@ambv
Copy link
Contributor

ambv commented Sep 5, 2024

Thank you for the report and patch!

@ambv ambv closed this as completed Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) pending The issue will be closed if no feedback is provided topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ambv @itamaro @arhadthedev @ChuBoning