virtualenv with --system-site-packages breaks pip's build isolation

[pganssle]

It seems that if you create a virtualenv with --system-site-packages, the system packages (but not the user packages) will be on the PYTHONPATH in the PEP 517 isolated build environment that pip creates (it does not effect python -m pep517.build), and it seems they will be on there with higher precedence than the requirements installed in the build environment. It only affects pip >= 19.0.0.

The most common way I've seen this cause problems is with the new setuptools.build_meta:__legacy__ backend. Since the most recent version of pip requires a recent version of setuptools, if you have an older version of setuptools installed on the system, pip install will fail due to the missing setuptools.build_meta:__legacy__ backend. It is possible to reproduce this by crafting a deliberately bad package and installing it, then creating a wheel for a newer version of the package (which allowed me to test that this failure was actually introduced with pip==19.0.0), but for the MWE we can stick with setuptools.

To reproduce, create a package like this:

cd /tmp
mkdir demo
cd demo
mkdir badpkg

touch badpkg/pyproject.toml
echo 'from setuptools import setup; setup(name="foo", version="0.1.0")' \
    > badpkg/setup.py

Then install an older version of setuptools on your system (can't be --user or in the virtualenv), pip install 'setuptools < 40.8.0' (I did this in a pyenv environment). If your system already has an older version of setuptools on it, you're already good.

Next create a virtualenv with --system-site-packages and activate it:

virtualenv venv --system-site-packages
source venv/bin/activate

Finally try to install badpkg (pip wheel or pip install -t tmp also works to demonstrate the problem):

pip install ./badpkg

You should get a traceback the ends like this:

AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

At first I thought this was because the affected packages had too loose bounds on build-system.requires (e.g. requires=["setuptools"]) and that pip was failing to install a more recent version in the isolated environment, but this bug still occurs even if you specify "setuptools>=40.8.0", so I believe it's not a problem at install-time, it's a problem with the path resolution at build time.

At the moment it's unclear if this is a problem with pip or virtualenv, but since it does not affect python -m pep517.build, I'm reporting it here. It could be a combination of both.

CC: @gaborbernat

[pganssle]

Per my Bloomberg colleague @lkollar, it seems that the problem is that pip creates a custom sitecustomize.py file in order to inject itself into the build environment.

Looks to me like it's just blacklisting the system packages with distutils.get_python_lib, which doesn't include the system path added by virtualenv. Seems like pep517.envbuild.BuildEnvironment doesn't have this problem - maybe switch over to using that? Or was there a reason to do it this way?

[boegel]

I'm also seeing this problem, when installing cryptography 2.5 using Python 3.7.2, even though I'm using the latest setuptools (40.8.0).

What I don't understand is why I'm seeing this even though I'm using pip install --no-build-isolation .... How is that possible if the problem is related to pip's build isolation feature?

[boegel]

I should also mention that I'm not using virtualenv at all (although maybe somehow the cryptography installation procedure does that somehow?).

[pganssle]

@boegel Can you create a minimal-working example that reproduces this, and include your pip versions? I've never known anyone to be able to trigger this without virtualenv --system-site-packages.

[boegel]

@pganssle Does this suffice?

Note that this is a self-built Python 3.7.2 from source, in a custom location, not sure if that's relevant to the problem...

$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug

$ curl -OL https://files.pythonhosted.org/packages/69/ed/5e97b7f54237a9e4e6291b6e52173372b7fa45ca730d36ea90b790c0059a/cryptography-2.5.tar.gz

$ tar xfz cryptography-2.5.tar.gz

$ cd cryptography-2.5

$ pip install --prefix=$PWD --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/user/pip_debug/cryptography-2.5
    Preparing wheel metadata ... error
    Complete output from command /prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e:
    Traceback (most recent call last):
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e" failed with error code 1 in /tmp/pip-req-build-q9knj58j

This is on CentOS 7.6.1810, with only python 2.x installed on the system. We do have an old setuptools in the OS as well, but I think that one is irrelevant since that's only for Python 2.

$ rpm -qa | grep ^python[0-9]*-[0-9]
python-2.7.5-76.el7.x86_64
$ rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch

[boegel]

I should also mention that --no-use-pep517 fixes the problem for me.

[pfmoore]

pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

Looks like your pip (and your setuptools) have been installed as eggs, presumably using easy_install. This is obsolete, can you try with a copy of pip installed using pip?

[boegel]

@pfmoore Same problem when pip and setuptools were installed with pip, when using latest cryptography from PyPI. Note that there's no other Python 3 installation on the system (and hence no other setuptools either).

$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g

Works fine when using --no-use-pep517:

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...

[pfmoore]

module 'setuptools.build_meta' has no attribute 'legacy'

That indicates you're not picking up setuptools 40.8.0 - see here

Try running

import setuptools.build_meta
print(setuptools.build_meta.__legacy__)

You should get something like <setuptools.build_meta._BuildMetaLegacyBackend object at 0x000000000401B2E8>. If you're not, then there's something wrong with your system setuptools. If you are, your build environment is not picking up the same packages as your system Python - in spite of your assertion that there's only the one setuptools installed on your system :-(

Also, the fact that you're using --prefix may be relevant here - I don't know how that interacts (if at all) with build environments. Can you reproduce this issue without --prefix?

[gaborbernat]

@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don't think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.

[pfmoore]

if virtualenv inherits the system site package

Presumably when --no-build-isolation is specified? An isolated build ignores the existing installation.

(and that has a setuptoools before 40.8.0)

Well, the point of --no-build-isolation is that the user is agreeing to set up the correct build environment, so that seems like it's a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that's the case, and his setuptools doesn't contain build_meta.__legacy__, then that setuptools installation is somehow broken, because 40.8.0 does contain that backend.

My suspicion is that @boegel is actually mistaken, and there's another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there's a bug somewhere - but if that is the case, then we're currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that's sort of the definition of "isolated" :-) But I'm not sure how relevant that is in this case, where --no-build-isolation is set.

which is not the case at the moment

That's what @pganssle is saying in the original post in this thread, yes. But I'm responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using --no-build-isolation.

If @boegel's problem turns out to be real, then I don't think it's related to this one (there's too many differences in the steps to reproduce it). So it should probably be a separate issue. But I'd be interested to understand why he thinks it is the same issue...

[boegel]

Dug a little further, it looks it's indeed a case of a broken setuptools installation, in some sense...

First, I should clarify my statement w.r.t. not having another setuptools installed. I should clarify that I mean that there's no other setuptools installed for Python 3.x. There is one for Python 2 (which is available in the OS), I'm assuming that one will never be picked up by Python 3.

That said, I'm pretty sure the right setuptools is being picked up:

$ python -c "import setuptools; print(setuptools.__file__); print(setuptools.version.__version__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/__init__.py
40.8.0

Yet:

$ python -c "import setuptools.build_meta; print(setuptools.build_meta.__file__); print(setuptools.build_meta.__legacy__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/build_meta.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

So I do have setuptools 40.8.0, but no __legacy__ in setuptools.build_meta?!

This sheds some light on it:

ls -ld /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools*
drwxr-xr-x 6 myuser myuser 4096 Mar  7 18:48 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:56 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.6.2.dist-info
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:57 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg-info
drwxr-xr-x 3 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm-3.2.0.dist-info

The setuptools-40.6.2.dist-info looks very suspicious, it seems like I somehow ended up with a frankenbuild of setuptools-40.6.2 (which doesn't have __legacy__ in setuptools.build_meta yet) and setuptools-40.8.0?! Ugh...

I got redirect to this issue via pypa/setuptools#1694, which shows the exact same error I'm running into, so I naively assumed it's the same problem.

It seems in my case it's indeed because of a corrupted setuptools though, my apologies for the noise...

[pfmoore]

No problem, glad we got to the bottom of it!

[Lawouach]

Hey,

I'm running into this issue as well and I'm thoroughly confused. As a mere user, it looks totally broken with no clear path of understanding how I'm doing anything wrong.

[Lawouach]

Well, having said that. I ended up removing the system-wide setuptools package and now local pip doesn't complain.

[tgs]

I'm encountering the same issue, I think. My organization defaults virtualenvs to --system-site-packages, and the globally installed setuptools is 40.4.3. I cloned the pip git repo, and then:

In a system-site-packages virtualenv, pip install ./pip/ gives the AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'.

In a virtualenv with no-site-packages, pip install ./pip/ works correctly.

I think I agree with @gaborbernat that it would be better for "build isolation" to include ignoring --system-site-packages. Or maybe there's at least some way to give a better error message?

[webknjaz]

FTR I'm hitting this with tox + system-site-packages with setuptools>=40.8.0 in pyproject.toml under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing pyproject.toml and then just playing with --no-build-isolation I've found this one...

Couldn't figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down...

[webknjaz]

Okay, now I'm hitting this in readthedocs/readthedocs.org#5572 somehow.

[webknjaz]

They use explicit --no-site-packages: https://readthedocs.org/projects/octomachinery/builds/8869252/

[webknjaz]

@pganssle it looks like it's time to remove --system-site-packages from the title. I've already hit two instances of the bug with --no-site-packages.

[slonopotamus]

I believe pypa/setuptools#3291 is another kind of this issue. Note that it neither involves virtualenv, nor Debian/Centos, but sympthoms are exactly the same: system-wide older version of setuptools has higher priority than one that is requested by pypackage.toml.

[khiz678]

Inside a Debian GNU/Linux 10 (buster) Docker container we create a virtual-env with --system-site-packages
Inside we have pip == 19.2.1
We are using pyproject.toml, requirements.txt and setup.cfg and install the latest setuptools == 63.2.0 using the pyproject.toml

We have the following code in the setup.cfg:

[options]
packages = find:
install_requires = file: requirements.txt

Now we hit the error where the get_requires_for_build_wheel fails with parsing error of the above setup.cfg.

File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
      requirement_string[e.loc:e.loc + 8]))
      pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "': requir'"

Upon inspection we found out that pip instead of loading the latest setuptools from the build environment loaded the setuptools == 40.8.0 from dist-packages and used the build_meta from
/usr/lib/python3/dist-packages/setuptools/build_meta.py

For those struggling with same bug using pip and venv on Debian distros, the current workaround we are using is to either uninstall the python3-setuptools or try to use the --no-build-isolation flag with pip

[AkechiShiro]

Is there anyone working on a fix for this issue it seems to be open for awhile, is there any relevant PR to fix issues here, it doesn't look like easy to provide a minimal working example tho.

I'm on ArchLinux and have encountered this issue btw while installing/adding Pendulum to my dependencies using Poetry.

[pfmoore]

@AkechiShiro not that I know of. The comment above from @IRDonch suggests a possible fix, but we'd need a test, so working out how to reproduce this is still a blocker here (as is someone actually preparing a PR for it, I don't believe any of the pip developers are working on it).

[AkechiShiro]

Thanks @pfmoore, could anyone guide me where I could try and dev this fix, in which folders is the buggy code, in the coming months, I hope I can get to fix this, I'm just not very knowledgeable with the code base, it's my first time working on pip, I'll be looking at the CONTRIBUTING.md probably present in the repo.

[SpecLad]

The comment above from @IRDonch suggests a possible fix

FYI, I tried to implement this fix, but I was blocked by #8273. virtualenv 19 includes a custom site module, which doesn't implement the getsitepackages function.

[minrk]

FWIW, this appears to affect mac system Python (macOS 12.6, Python 3.9.6) as well.

pyproject.toml

[build-system]
requires = [
    "setuptools>=60",
]
build-backend = "setuptools.build_meta"

[project]
version = "0.1.0"
requires-python = ">=3.7"
license = {text = "MIT License"}
name = "debug"
description = "..."

setup.py

from setuptools import setup
import setuptools
import sys

print("\n\n")
print(f"in setup.py: {setuptools.__version__=}, {setuptools.__file__=}")
path_lines = '\n  '.join(sys.path)
print(f"sys.path=\n  {path_lines}")
print("\n\n")

setup()

pip list

$ /usr/bin/python3 -m pip install --upgrade --user setuptools pip
Requirement already satisfied: setuptools in /Users/minrk/Library/Python/3.9/lib/python/site-packages (65.3.0)
Requirement already satisfied: pip in /Users/minrk/Library/Python/3.9/lib/python/site-packages (22.2.2)
$ /usr/bin/python3 -m pip list
Package    Version
---------- -------
altgraph   0.17.2
future     0.18.2
macholib   1.15.2
pip        22.2.2
setuptools 65.3.0
six        1.15.0
UNKNOWN    0.0.0
wheel      0.37.0

Building wheel with pip:

$ /usr/bin/python3 -m pip wheel -v .
Processing /private/tmp/test-package
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=60
    Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-65.3.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.37.1
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info/SOURCES.txt'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml): started
  Running command Building wheel for UNKNOWN (pyproject.toml)



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running bdist_wheel
  running build
  running install
  running install_egg_info
  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Copying UNKNOWN.egg-info to build/bdist.macosx-10.9-universal2/wheel/UNKNOWN-0.0.0-py3.9.egg-info
  running install_scripts
  Building wheel for UNKNOWN (pyproject.toml): finished with status 'done'
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=964 sha256=75197df67b739145c46f3017b89c662f98118e90f27a93a29a356b76b7b10af0
  Stored in directory: /Users/minrk/Library/Caches/pip/wheels/63/83/5c/e98bf9706ef6a83be6f3ec105be9771fbd8f4d9d70f9257288
Successfully built UNKNOWN

And again without build isolation, fixes the issue:

$ /usr/bin/python3 -m pip wheel -v --no-build-isolation .
Processing /private/tmp/test-package
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/Users/minrk/Library/Python/3.9/lib/python/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Users/minrk/Library/Python/3.9/lib/python/site-packages
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info
  writing /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/top_level.txt
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug-0.1.0.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: debug
  Building wheel for debug (pyproject.toml): started
  Running command Building wheel for debug (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/Users/minrk/Library/Python/3.9/lib/python/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Users/minrk/Library/Python/3.9/lib/python/site-packages
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages



  running bdist_wheel
  running build
  installing to build/bdist.macosx-10.9-universal2/wheel
  running install
  running install_egg_info
  running egg_info
  creating debug.egg-info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  writing manifest file 'debug.egg-info/SOURCES.txt'
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Copying debug.egg-info to build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0.dist-info/WHEEL
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-wheel-buzyl83a/tmped7n05ks/debug-0.1.0-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-universal2/wheel' to it
  adding 'debug-0.1.0.dist-info/METADATA'
  adding 'debug-0.1.0.dist-info/WHEEL'
  adding 'debug-0.1.0.dist-info/top_level.txt'
  adding 'debug-0.1.0.dist-info/RECORD'
  removing build/bdist.macosx-10.9-universal2/wheel
  Building wheel for debug (pyproject.toml): finished with status 'done'
  Created wheel for debug: filename=debug-0.1.0-py3-none-any.whl size=957 sha256=8c9826d6865bb428a68051a817b34b1f87844cf9fa367b8379674d1fca9c381e
  Stored in directory: /Users/minrk/Library/Caches/pip/wheels/63/83/5c/e98bf9706ef6a83be6f3ec105be9771fbd8f4d9d70f9257288
Successfully built debug

EDIT: running in a venv _does_ help

$  /usr/bin/python3 -m venv /tmp/debug-env
$ source /tmp/debug-env/bin/activate
(debug-env) $  python3 -m pip wheel -v .
Processing /Users/minrk/dev/py/6d73fc9159448f1f83f8a6d5c2b2dfac
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=60
    Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-65.3.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running egg_info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.37.1
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info
  writing /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/top_level.txt
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug-0.1.0.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: debug
  Building wheel for debug (pyproject.toml): started
  Running command Building wheel for debug (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running bdist_wheel
  running build
  installing to build/bdist.macosx-10.9-universal2/wheel
  running install
  running install_egg_info
  running egg_info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Copying debug.egg-info to build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0.dist-info/WHEEL
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-wheel-9il5vyyn/tmp6zqsfanh/debug-0.1.0-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-universal2/wheel' to it
  adding 'debug-0.1.0.dist-info/METADATA'
  adding 'debug-0.1.0.dist-info/WHEEL'
  adding 'debug-0.1.0.dist-info/top_level.txt'
  adding 'debug-0.1.0.dist-info/RECORD'
  removing build/bdist.macosx-10.9-universal2/wheel
  Building wheel for debug (pyproject.toml): finished with status 'done'
  Created wheel for debug: filename=debug-0.1.0-py3-none-any.whl size=959 sha256=b5f9cb097a864dd577a7cea3f7805b913d1968bbad235754d9d0293ded1e361c
  Stored in directory: /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-ephem-wheel-cache-qx6qpam5/wheels/0b/37/75/5000e2f11e9ea9460d2d869496734ee8dffb476f3090f04947
Successfully built debug

[minrk]

Minimal repro with docker on ubuntu:22.04: https://gist.github.com/minrk/6d73fc9159448f1f83f8a6d5c2b2dfac

[fredthomsen]

I am seeing the system-wide paths have the wrong precedence: /usr/lib/python3.6/site-packages/* take precedence over the /usr/local/lib/python3.6/* for building my package for a tox environment in this case. Does that fall in line with what others are seeing?

[minrk]

It seems to me that the simplest fix would be to remove all directories in site.getsitepackages() from sys.path and not just platlib/purelib.

#11466 uses site.getsitepackages() if available, and falls back on the current behavior in the case where it isn't (i.e. virtualenv<20). I've confirmed at least that it fixes the issue I'm seeing on system Python in Ubuntu and macOS.