{Core} aaz: Fix importlib deadlocks while loading requests.models in multi threads
#26287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
️✔️AzureCLI-FullTest
|
azure-client-tools-bot-prd
bot
added this to the May 2023 (2023-05-23) - For Build (Breaking change release) milestone
ghost
added
ghost
requested a review
from 
|
Core |
jiasli
reviewed
Apr 28, 2023
| @@ -12,6 +12,10 @@ | |||
| from azure.core.polling.base_polling import LROBasePolling | |||
| from azure.core.tracing.common import with_current_context | |||
| from azure.core.tracing.decorator import distributed_trace | |||
| # import LROPoller in main thread to resolve import deadlock between threads in python 3.10.10 | |||
| # reference https://github.com/psf/requests/issues/2925 and https://github.com/Azure/azure-cli/issues/26272 | |||
| from msrest.polling.poller import LROPoller # pylint: disable=unused-import | |||
There was a problem hiding this comment.
My gut feeling tells me this may not be safe enough, as there may be other dependencies imported by the sub-thread and main thread at the same time.
The SDK implementation is safer because the first PUT operation is performed on the main thread, so subsequent polling GET operations on the sub-thread will have the full dependency tree cached already.
Besides LROPoller, there are also AzureOperationPoller and AzureCoreLROPoller imported from msrestazure.azure_operation and azure.core.polling respectively:
azure-cli/src/azure-cli-core/azure/cli/core/commands/__init__.py
Lines 1153 to 1157 in d9accde
azure-cli/src/azure-cli-core/azure/cli/core/util.py
Lines 685 to 690 in dae19d0
Though I haven't got time to look into their actual usage.
There was a problem hiding this comment.
This line is enough to mitigate the issue. Moving the first request in the main thread will involve a huge change.
We cannot resolve them all because it's python issue python/cpython#91351.
There was a problem hiding this comment.
When loading other modules, some of them will import requests. That's why it can mitigate the issue.
necusjz
approved these changes
May 8, 2023
yash-nisar
pushed a commit
to yash-nisar/azure-cli
that referenced
this pull request
May 16, 2023
… in multi threads (Azure#26287) * import LROPoller in main thread to resolve importlib deadlock in python 3.10.10 * use 'import requests' * fix style issue * update comments
avgale
pushed a commit
to avgale/azure-cli
that referenced
this pull request
Aug 24, 2023
… in multi threads (Azure#26287) * import LROPoller in main thread to resolve importlib deadlock in python 3.10.10 * use 'import requests' * fix style issue * update comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closed: #26272
closed: #26297
Related command
Description
In python 3.10.10, we encountered import deadlocks in
aazLRO which try to load therequests.modelsin two different threads. It's a known issue psf/requests#2925In this PR, we provide a workaround by importing these modules in the main thread before spawning child threads.
Testing Guide
History Notes
[Component Name 1] BREAKING CHANGE:
az command a: Make some customer-facing breaking change[Component Name 2]
az command b: Add some customer-facing featureThis checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.