-
-
Notifications
You must be signed in to change notification settings - Fork 30.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Py_GIL_DISABLED
is not set on Windows
#127294
Comments
This is mentioned in https://docs.python.org/3/howto/free-threading-extensions.html#windows. That text was added in 02b272b, which mentions was based on https://github.com/Quansight-Labs/free-threaded-compatibility/, but I was unable to find any references to this Windows quirk there, so I don't know if there was any prior discussion about it. @colesbury, perhaps you could help clarify if there was any prior discussion? I think it's important to understand why this limitation exists, and what we could possibly do to get around it. |
That doesn't seem great 🫤 From what I can conjecture from #115582 (comment), it seems like we have a single installer that installs both free-threaded and regular binaries under the same prefix ( The approach we have taken with Python installations on Windows is that each installation must have its own prefix. On POSIX, the prefix can be shared, as all paths containing installation-specific files include the version and ABI flags. If we want to fix this issue, I think free-threaded builds on Windows should either be installed to a different prefix, or adopt the POSIX approach 1. To keep in line with the usual expectations of how programs work on Windows, I think it's better to keep the paths as-is, and simply install free-threaded builds to a different prefix. Steve mentions the issue with that approach:
I don't think I understand which specific use-case this is referencing. @zooba could you clarify? Footnotes
|
I am not prepared to produce a second set of installers in order to install to a separate prefix, nor do I want to modify the existing installer in order to do that. It is too big a risk for regular users. I presented this to the steering council and they agreed with the current design. We also don't want to confuse users further by having a much longer list of installers to choose from on the downloads page. One day, the separate free-threaded builds go away and we'll just have one install again (potentially with free-threaded enabled all the time by default). I don't care to over-design this scenario now, since we know it'll go away. POSIX is different because a build-time generated So basically, we need an idea that:
My best recommendation right now is to get your freethreaded build through uv, which ought to be getting it from the Nuget packages, which already meet all of these requirements. For those who'd rather get their builds from Nuget directly (like cibuildwheel, IIUC), that's also pretty straightforward. But I really don't want to mess with the installer too much. It's fragile enough already (I'd prefer to deprecate and replace it entirely, but haven't got a better option - though I'm starting to discuss better options). |
There's more context on why the current design is as it is at python/steering-council#225 (with links to even more context) |
That's fair, though it's not clear to me how installing to different prefixes would add a significant risk for users.
Sorry I missed that, though reading the discussion there, it doesn't seem like the implications of having different build in the same prefix were properly considered.
I agree, I think the best option would be to modify the current installer to install both builds in a separate prefix, but if you aren't comfortable with that, that's okay.
Yes, but the PEP 703 estimate for that is 2026–2027, so it'll still be a while. But I understand where you are coming from.
I don't understand how that affects things here 😅
My proposal was to have the installer install a regular build to Otherwise, the only other option I see is to add the version and ABI flags to the paths, like it's done for POSIX. |
Yeah, it's really not obvious until you get pretty deep into Windows Installer (and have debugged a number of weird scenarios). Basically it boils down to the installers being a mix of generated, calculated, and static UUIDs. These identify installable products, features and components, which may contain files. The files themselves don't matter to the installer, just the UUID. Which means if you get any unintentional overlap, you get corrupted installs - files won't be properly added, or they'll be removed or moved to a different location. The advantages are all for in-place updates - we can reliably detect previous versions of the same thing, and handle them efficiently. It doesn't matter so much these days, because we no longer care about the difference between downloading 100KB vs 1MB, or copying 100 files vs 1000 files. So it seems just as efficient to just download/copy everything all the time, and the effort to minimise things seems wasted. But anyway, with all these UUIDs around, forking the installer to be parallel to the original requires very carefully making sure all the static UUIDs are new, and all the calculated ones aren't going to generate conflicts. Otherwise, users who have a regular install and the freethreaded install will get weird behaviour - weird as in individual files just disappearing, or install steps being skipped no matter how you try to trigger them, or entire features being skipped because they're "already installed". We recently had a release refuse to "downgrade" a component, and so it just removed it and silently left it missing, and people's Python wouldn't run unless they'd installed an equivalent component separately. And this is before we get to things like registry keys or environment variables. Like I said, I'd love to deprecate and replace the whole thing. But I won't do that without an equally user friendly alternative, because literally millions of people rely on clicking through this on a regular basis. I've been thinking about/designing alternatives for years, and am only just starting to get towards something I'm happy with, but we're a few years away from being able to deprecate this current installer. Free threading will likely be merged by then ;) |
Gotcha, I understand now. What about the alternative, then? Changing the install layout to include the version and ABI flags. |
From the installer POV, that's just a different default install directory. That's the trivial part - it's being able to install the second installer at the same time as the main one and have them both work correctly that adds all the risk. I suspect the people who are trying to validate free-threaded binaries at this point would be just as happy to download a zip file and manage it themselves? If so, they can go to python-freethreaded, pythonx86-freethreaded or pythonarm64-freethreaded and use the "Download package" link on the right to get them (it's a zip file renamed to .nupkg, and the binaries are under the The challenge is the install method, not the layout. |
Ah, right 🤦 Other thought, would changing between the regular and free-threading binaries based on a user selection in the installer be viable? This would add the limitation of only allowing either one, but I personally think that's okay.
I am happy with any of those scenarios, and would be +1 towards dropping the free-threaded support from the installer. Having two builds with the same paths is problematic, and I am sure we will see more issues other than the |
I wouldn't be sure of that at all, given that no other issues have materialized yet and testing by package authors and end users has been pretty heavy recently. Dropping the option for what is at this point a hypothetical second issue does not seem like a good idea, since it incurs a bunch of work for docs, breaks blog posts already published, and may break CI systems that rely on scripts pointing at the current installers (e.g., if using the Doing nothing and closing this issue seems preferable here, unless more bugs show up. |
The current user base is small and technical, I am not worried about those users. As this gets further adoption, less technical users will start experimenting with it and will eventually run into issues they absolutely shouldn't, like mismatched package installations. Sharing ffy00@DESKTOP-VP183NR C:\Program Files\Python313>python3.13t -m pip install numpy
Collecting numpy
Downloading numpy-2.1.3-cp313-cp313t-win_amd64.whl.metadata (60 kB)
Downloading numpy-2.1.3-cp313-cp313t-win_amd64.whl (12.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.6/12.6 MB 3.8 MB/s eta 0:00:00
Installing collected packages: numpy
WARNING: The scripts f2py.exe and numpy-config.exe are installed in 'C:\Program Files\Python313\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-2.1.3
[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python3.13t.exe -m pip install --upgrade pip
ffy00@DESKTOP-VP183NR C:\Program Files\Python313>python
Python 3.13.0 (tags/v3.13.0:60403a5, Oct 7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "C:\Program Files\Python313\Lib\site-packages\numpy\_core\__init__.py", line 23, in <module>
from . import multiarray
File "C:\Program Files\Python313\Lib\site-packages\numpy\_core\multiarray.py", line 10, in <module>
from . import overrides
File "C:\Program Files\Python313\Lib\site-packages\numpy\_core\overrides.py", line 8, in <module>
from numpy._core._multiarray_umath import (
add_docstring, _get_implementing_args, _ArrayFunctionDispatcher)
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python313\Lib\site-packages\numpy\__init__.py", line 127, in <module>
from numpy.__config__ import show as show_config
File "C:\Program Files\Python313\Lib\site-packages\numpy\__config__.py", line 4, in <module>
from numpy._core._multiarray_umath import (
...<3 lines>...
)
File "C:\Program Files\Python313\Lib\site-packages\numpy\_core\__init__.py", line 49, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.13 from "C:\Program Files\Python313\python.exe"
* The NumPy version is: "2.1.3"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy._core._multiarray_umath'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
import numpy
File "C:\Program Files\Python313\Lib\site-packages\numpy\__init__.py", line 132, in <module>
raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.
>>> And since pretty much all projects ship separate wheels for free-threaded and non-free-threaded builds, making it impossible to have a package installation that is valid in both builds, as it stands you pretty much have to decide whether you want a free-threaded or non-free-threaded environment. While I think this is okay while we are still working on the free-threaded adoption by the bigger packages, as soon as that is done, and we start getting the broader community to test free-threading and adding support to their own projects, I personally don't think this UX is acceptable.
You are right in pointing out the impact of changes in the installer, and that should be a consideration while deciding how to proceed, but I disagree with the severity of the issue. Like I said, I don't think it's a major issue right now, but I think it will be as free-threading starts seeing more adoption.
I disagree with you on this 🫤 I think if it's possible to select between the free-threaded and regular build on the installer, we should consider that. I'll wait for Steve to say if that's viable or not. If that is not possible, I think we should, at the minimum, add the ABI flags to the site-packages path on Windows. |
We can make changes for 3.14, no problem. It's just deciding what those changes will be. Having the installer switch is basically as bad as having two parallel installers, or alternatively it's confusing in that Merging the Alternatively, we take it out of the installer entirely and figure out some reasonable way to link to three ZIP files (or three "cheap" installers) from the downloads page? Some way that ensures the 99% of downloads who don't want it won't choose it by mistake - the existing six entries on there are already confusing enough.
I wish this pointed to the Nuget packages instead :( |
We're also way off the topic of IMHO, I really don't like dynamically generating header files at build time, and would much prefer build tools determine whether they want this option themselves. If that's really not going to work, then I'd prefer to generate it at installer build time (or perhaps just select it), which would mean that in-tree builds get harder to work with (compared to the build tool setting it; not compared to compile-time generation). |
Happy to make that change. Do you have a preferred invocation (I don't know much about Nuget, sorry)? |
You can stick with the Users who already know Nuget may choose to use that themselves. It doesn't do version resolution - you have to pin stuff - so the difference between |
Is there a way to register two
The package installers should be getting data from With GH-112984, venv will only copy the correct Python, but I am not sure if virtualenv and other environment managers will do the same, so that's a possible issue. If you want to focus on the venv-only use-case and don't want to rename site-packages, we could possibly add a PEP 668
Yeah, I feel like the current UI isn't cohesive for experimental releases like these. Since this isn't urgent, we could try asking the PSF if there are any resources to help us do a small redesign of the download pages? I would not be opposed to do that work myself, but I don't think I am knowledgeable enough in UI/UX to do a good job 😅. We could also do some outreach and see if any community members experienced in UI/UX would be willing to help? Do you think it makes sense to go back to the SC for a new opinion?
It's related, as that issue is caused by sharing the prefix between builds, but maybe we should move it. |
When there's a new SC, yeah, it won't hurt. But they'll want no more than two options to choose between, so we should come up with concrete proposals.
It's also caused by us not properly generating headers at compile time, which is something that we've never done here before and I would prefer to undo the hack that we have. So either we're going to drastically change part of the Windows build (and significantly break/upset in-tree builds), or we push that generation out to install time (and break experimental build in-tree users), or we push the option to sysconfig/the build tools and everyone is happy. |
Just a note that it's pretty common to simultaneously have all the python versions needed for testing or wheel building simultaneously loaded on CI. Not being able to do this, but only for free threading on windows, will be surprising. |
Bug report
Bug description:
From pypa/distutils#310
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
The text was updated successfully, but these errors were encountered: