Free-threaded libraries should be in lib/python3.14t (configure)

[colesbury]

Bug report

Background

When using configure based Python installations, if two the free-threaded and default builds are installed to the same prefix, they will share the same lib directory. For example, when installing Python 3.14 the structure looks like:

include/python3.14/...
include/python3.14t/...
lib/libpython3.14.a
lib/libpython3.14t.a
lib/python3.14/...  # shared!

The include directories are not shared, which is good because they have different pyconfig.h files.

However, the lib/python3.14 is shared, which means that packages installed in the default build may be partially available in the free-threaded build and vice versa. This was unintended and can cause problems, such as confusing error messages and crashes.

For example, if I run:

• python3.14 -m pip install numpy (install in default build)
• python3.14t -c "import numpy"

I get a confusing error message:

Error importing numpy: you should not try to import numpy from its source directory...

Traceback (most recent call last):
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/_core/overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
        add_docstring,  _get_implementing_args, _ArrayFunctionDispatcher)
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show as show_config
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
    ...<3 lines>...
    )
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/_core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.14 from "/tmp/python-nogil/bin/python3.14t"
  * The NumPy version is: "2.0.0"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy._core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import numpy
  File "/tmp/python-nogil/lib/python3.14/site-packages/numpy/__init__.py", line 119, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

It would be better if installing NumPy in the default build did not make it available in the free-threaded build and vice versa.

Proposal

We should add the ABI suffix to the lib directory, like we do for the include directory. Specifically, we should use python$(LDVERSION) instead of python$(VERSION).For example, the free-threaded build would use lib/python3.14t in Python 3.14.

Debug builds have d as part of their ABI suffix (and LDVERSION), so this would incidentally affect installations of the debug configuration of the non-free-threaded build.

Linked PRs

• gh-121103: Put free-threaded libraries in lib/python3.14t #121293
• [3.13] gh-121103: Put free-threaded libraries in lib/python3.14t (GH-121293) #121631
• gh-121103: Update site module docs for free-threaded installs #122737
• [3.13] gh-121103: Update site module docs for free-threaded installs (GH-122737) #122750

[encukou]

Debug builds have d as part of their ABI suffix (and LDVERSION), so this would incidentally affect installations of the debug configuration of the non-free-threaded build.

That's the catch. The debug build is ABI-compatible; you do want this kind of cross-installation for the debug build.

Perhaps there needs to be a new variable with just the t flag :/

[colesbury]

Yeah, I think just the t flag is probably a good choice to avoid affecting the default build.

[asottile]

is there a reason to separate the stdlib as part of this? as far as I understand the stdlib would be byte-identical (ignoring the .pyc files -- but those have the api/abi parts in them already) between the free-threaded and classic build and not subject to the issues highlighted here.

from this issue it seems to me that this is only an issue for site-packages and having a duplicate lib / lib-dynload for the stdlib feels like an unfortunate side-effect

with the duplicated copy this complicates distribution packaging (at least on debianlikes) where the stdlib is shipped in several packages (to separate minimal files, arch independent files, etc.). I'm inclined to undo that part of this patch to unbreak the builds and simplify things but I'm hesitant to deviate with no hope of convergence

I'm also not quite sure what to do since this is a pretty significant breakage late in the beta period -- fortunately I'm noticing this due to nightly builds!

[asottile]

at least for past precedence -- prior to the debug builds becoming abi-compatible this wasn't considered an issue (though the same exact problem occurred there!). I suspect (and maybe I'm wrong) that this is considered higher priority because free-threaded is shiny-new :)

maybe we don't need this change at all and can instead recommend virtualenvs? I don't think mixing pythons in a virtualenv makes sense anyway

[itamaro]

is there a reason to separate the stdlib as part of this? as far as I understand the stdlib would be byte-identical (ignoring the .pyc files -- but those have the api/abi parts in them already) between the free-threaded and classic build and not subject to the issues highlighted here.

You can have different patch versions installed side-by-side (i.e., 3.13.1 & free-threaded-3.13.3) with a common prefix - in that case the stdlib py files are not identical and will lead to confusing and difficult to debug behaviors.

[colesbury]

is there a reason to separate the stdlib as part of this?

As I wrote on the PR, my concern is that sharing .py files would allow for situations where the stdlib .py files don't match the .so files. That's might not be a concern for package-based installations, but it would likely come up for ./configure based installs.

... this complicates distribution packaging (at least on debianlikes) where the stdlib is shipped in several packages

Would it make sense to override the _INSTALL_SCHEMES in that case?

... prior to the debug builds becoming abi-compatible this wasn't considered an issue... maybe we don't need this change at all and can instead recommend virtualenvs?

We prioritize this because we got multiple bug reports related to this problem, and the way these failures manifest is confusing for even experienced CPython developers.

The abi-incompatible debug builds did not make for a good user experience, and I don't think we want to emulate them.

[asottile]

is there a reason to separate the stdlib as part of this? as far as I understand the stdlib would be byte-identical (ignoring the .pyc files -- but those have the api/abi parts in them already) between the free-threaded and classic build and not subject to the issues highlighted here.

You can have different patch versions installed side-by-side (i.e., 3.13.1 & free-threaded-3.13.3) with a common prefix - in that case the stdlib py files are not identical and will lead to confusing and difficult to debug behaviors.

I don't think this is helpful to this discussion: installing 3.12.1 and 3.12.2 side-by-side will not work -- they will clobber each other's stdlib (the patch version isn't part of the libdir)

[colesbury]

The case @itamaro and I are referring to, is when you install 3.12.1 default and 3.12.2 free-threaded side by side. If you share .py files from the stdlib (but not .so files from the stdlib), then you get a partial clobber of the co-located installation, which is bad.

[asottile]

Would it make sense to override the _INSTALL_SCHEMES in that case?

yes that's precisely where the merge conflict I'm dealing with is right now (since debian also does the whole dist-packages / usr/local dance as well in this same mapping) -- I'm just worried that diverging here (by not making the stdlib separate) will lead to further breakage down the line. I'm advocating for cpython also keeping them the same and only diverging for site-packages (where the demonstrated breakage occurred)

[asottile]

The case @itamaro and I are referring to, is when you install 3.12.1 default and 3.12.2 free-threaded side by side. If you share .py files from the stdlib (but not .so files from the stdlib), then you get a partial clobber of the co-located installation, which is bad.

I understand that, but I assert that nobody would want that (or we would already encode the patch version in the libdir) -- in the same way that they wouldn't want to install two patch versions side-by-side in the same prefix independent of free-threaded

edit emphasized

[colesbury]

yes that's precisely where the merge conflict I'm dealing with is right now...

I think I understand your concerns in general, but I'm unsure about the specifics. As I understand it they are:

1. Concern that future changes will re-break deadsnakes (and other distributors), given that this broke the nightly builds
2. Concern that future changes will entrench the notion of a split stdlib between the default and free-threaded builds in a way that can't be worked around by distributors.

On (1), I think there is still some work on duplicate symlinks, but I think that will be a smaller change.

On (2), I think we still want to support the way Debian, deadsnakes, Fedora, etc. distribute Python, and that includes configurations that share the stdlib.

For specific changes, I'm looking at https://github.com/deadsnakes/python3.13/blob/ubuntu/jammy/debian/patches/distutils-install-layout.diff. It seems like the patch needs to be refreshed because the base changed, but is otherwise fine. In particular, it looks like Debian does not depend on CPython's posix_prefix scheme, but instead uses posix_local or `deb_system, both of which the patch defines. Is that correct?

I understand that, but I assert that nobody would want that...

It's not a matter of people deliberately wanting two different patch versions. If you are upgrading a co-located local configure based installation, you will likely end up in this situation because make install will only upgrade one or the other and people are not likely to remember to upgrade both the default and free-threaded installations at the same time.

People are definitely doing this now, including CPython core developers.

At one point previously, I suggested that people should install the default and free-threaded builds in separate prefixes, but:

1. That suggestion was not well received by other core developers
2. Based on bug reports, it's become clear to me that people would not adhere to that suggestion

It would have been better if we addressed this earlier in the development cycle, but it still seems better to do this now (before beta 4), rather than leave the problem for later.

[asottile]

getting back around to this I'm having to also revert the patch to configure and Makefile.pre.in making me much more nervous about carrying these patches forever

[asottile]

seems like this patch also breaks noarch conda packages per the linked issue ^^