gh-91351: Support re-entrancy in importlib/_bootstrap.py #94342
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a PR against the 3.9 branch (I will forward-port changes after addressing other review feedback).
Note that most of the size of the diff for this PR is generated changes to re-freeze
_bootstrap.py
and most of what's left after subtracting that is new comments about how the implementation works to make it easier to understand.See #91351 for details about the problem.
Re-entrancy is always tricky and given the requirements of _bootstrap.py (to operate with re-entrancy and multi-threading and to do so without exposing any of the details to application code doing an import) I think this goes double.
This PR does a few things to achieve better safety in the face of re-entrancy:
I'm not quite sure I believe this new version of the code is 100% correct with respect to re-entrancy but it does fixes mishandling of two specific cases:
_blocking_on
is populated and cleaned up inside_ModuleLock.acquire
. Previous this failed with aKeyError
(as described in the linked issue)._ModuleLock.acquire
. Previously this failed by deadlocking.This PR also does not include any new unit tests. I have a small stand-alone program which can reproduce both of these but only with the assistance of some additional instrumentation inside
_bootstrap.py
to make sure the re-entrancy happens at the interesting times. If adding this kind of instrumentation is acceptable then it may be possible to turn this program into some unit tests.It may also be possible to simplify
_BlockingOnManager
by switching_ModuleLock.lock
to anRLock
. That solution didn't originally occur to me so I developed this - but if others think that is a better approach I think it's a fairly simple change.For reference, here is a stand-alone reproducer. This one isn't quite deterministic but by running the codepath over and over it seems to be fairly reliable in reproducing one of the problem codepaths on my system. For a completely deterministic reproducer, I think _bootstrap.py instrumentation is required.