Eliminate conflicts between build variants installed to a common prefix

[ned-deily]

When using configure-based (i.e. non-Windows) Python installations, if more than one build variant is installed to the same prefix, some files in the bin and lib/pkgconfig directories currently are installed with the same name by each variant, with potentially unpredictable and undesired results.

As of Python 3.13, there are currently four build variants of interest:

• default (no suffix)
• debug (d suffix)
• free-threaded (t suffix)
• free-threaded debug (td suffix)

Issue #121103 and associated PRs implemented a separation of the installed lib/python3.x directory into separate directories when installing to a common installation prefix: lib/python3.x (for the default and debug variants) and lib/python3.xt (for the free-threaded and free-threaded debug variants). The rationale was primarily to avoid confusion when installing a third-party package that only supports one variant but is on the import path of any variant due to the common directories on sys.path.

However, there still remain conflicts elsewhere within the common install prefix, namely, in the bin and lib/pkgconfig directories.

When more than one variant shares a bin directory, there are two kinds of conflicts: (1) files installed by the cpython build system with the same name but variant-specific content; and (2) script files (or any other file) installed by a third-party package to the bin directory.

For the second category, the main issue is that, if the user wishes to install a third-party package to more than one variant, its scripts will only be associated with one instance (through its shebang line). There are straight-forward workarounds for this, like installing in separate venvs, if the user is aware of them, but those solutions might not always be desirable. Perhaps the best we can do for this at the moment is to try to document best practices. This is not further addresses here.

The first category of conflicts are centered around the cpython Makefile. For python unix (configure / Makefile) installs, there are two types of installs: make altinstall which installs bin files with a 3.x suffix, i.e. python3.13; and make install which also installs unversioned links as well as those installed by altinstall, i.e. python3 along with python3.13. There are also similarly-versioned .pc files installed in lib/pkgconfig for use by third-party pkg-config utilities.

Since free-threading as introduced in Python 3.13 is considered an experimental feature, it was decided to keep free-threaded and default (traditional GIL-enabled) builds independent initially, in particular, by using different names to invoke the different interpreters, python3.13 vs python3.13t, without doing anything to preclude moving to a single build in a future feature release. While the safest way to keep them independent would be to install each to a separate prefix, like /opt/python3.14 vs /opt/python3.14t, there are advantages to and expectations by users and distributors to using a common install prefix for each variant, say /usr/local/bin/python3.13 and /usr/local/bin/python3.13t and possibly /usr/local/bin/python3.13td.

As it stands now, though, there are a number of conflicting file names that are getting installed by each variant. Here is an exhaustive list of conflicts:

by make altinstall

    bin/
        idle3.x
        pip3.x
        pydoc3.x
        python3.x
    lib/
        pkgconfig/
            python-3.14-embed.pc
            python-3.14.pc

by make install (in addition to those of altinstall)

    bin/
        idle3
        pip3
        pydoc3
        python3
        python3-config
        python3.x-config
    lib/
        pkgconfig/
            python3-embed.pc
            python3.pc

To resolve this, I recommend the following changes:

• no changes to a default build install/altinstall.
• make altinstall for d, t, and td variants
  • do not install python3.x and python3.x-config
  • install idle3.x, pydoc3.x as idle3.x{v}, pydoc3.x{v}
  • install pip3.x as pip3.x{v} -> requires change in pip and/or ensurepip?
  • do not install lib/pkgconfig python-3.x-embed.pc and python-3.x.pc
• make install for d, t, and td variants
  • install python3, python3-config as python3{v}, python3{v}-config
  • install idle3, pydoc3 as idle3{v}, pydoc3{v}
  • install pip3 as pip3{v} -> requires change in pip and/or ensurepip?
  • install lib/pkgconfig python3-embed.pc, python.pc as python3{v}-embed.pc, python3.pc

The associated PR implements these changes.

As an example,
./configure && make && make install

produces the following (both before and after this PR with -> indicating a symlink):

bin
    idle3.13
    idle3 -> idle3.13
    pip3.13
    pip3
    pydoc3.13
    pydoc3 -> pydoc3.13
    python3.13
    python3 -> python3.13
    python3.13-config
    python3-config -> python3.13-config
lib/pkgconfig/
    python-3.13.pc
    python3.pc -> python-3.13.pc
    python-3.13-embed.pc
    python3-embed.pc -> python-3.13-embed.pc

while ./configure --disable-gil && make && make install
produces this currently:

bin
    idle3.13
    idle3 -> idle3.13
    pip3.13
    pip3
    pydoc3.13
    pydoc3 -> pydoc3.13
    python3.13
    python3 -> python3.13
    python3.13t
    python3.13t-config
    python3.13-config -> python3.13t-config
    python3-config -> python3.13-config
lib/pkgconfig/
    python-3.13t.pc
    python-3.13.pc -> python-3.13t.pc
    python3.pc -> python-3.13.pc
    python-3.13t-embed.pc
    python-3.13-embed.pc -> python-3.13t-embed.pc
    python3-embed.pc -> python-3.13-embed.pc

and with the PR applied produces this:

bin
    idle3.13t
    idle3t -> idle3.13t
    pip3.13
    pip3
    pydoc3.13t
    pydoc3t -> pydoc3.13t
    python3.13t
    python3t -> python3.13t
    python3.13t-config
    python3t-config -> python3.13t-config
lib/pkgconfig/
    python-3.13t.pc
    python3t.pc -> python-3.13t.pc
    python-3.13t-embed.pc
    python3t-embed.pc -> python-3.13t-embed.pc

Note that the only overlapping files names with the PR applied are those of pip. The pip install itself creates those files, rather than the Python Makefile. Without delving into pip itself, I did not see an obvious fix. Perhaps a combination of pip and/or ensurepip changes may be needed.

Additional documentation for the above aimed at users and distributors is probably also warranted.

Also, it should be noted that framework builds (--enable-framework) on macOS and iOS have additional common prefix compatibility issues that are not addressed above. Fortunately, there is a straightforward workaround: use separate framework names for each variant (--with-framework-name=...). (The python.org macOS installers for 3.13 are using Python and PythonT for the default and the free-threading builds.) Further work on improving framework commonality may be the subject of a future issue and PR.

Linked PRs

• gh-122402: Eliminate conflicts between build variants installed to a common prefix #122403