Virtual environment breaks test_sysconfig and test_venv tests on macOS

[artemmukhin]

Bug report

1. Build CPython on macOS
2. Activate a virtual environment as described in https://devguide.python.org/testing/coverage/#install-coverage
3. Run python -m test

Results in:

== Tests result: FAILURE ==

2 tests failed:
    test_sysconfig test_venv

$ python -m test test_venv --verbose
...
AssertionError: Lists differ

First differing element 0:
'/.../bin/python.exe'
'/.../bin/python'

TODO

• Fix test_sysconfig gh-103224: Resolve paths properly in test_sysconfig #103292
• Fix test_venv GH-103224: Use the realpath of the Python executable in test_venv #103243

Your environment

• CPython versions tested on: main
• Operating system and architecture: macOS 13.2.1

Linked PRs

• GH-103224: Use the realpath of the Python executable in test_venv #103243
• gh-103224: Resolve paths properly in test_sysconfig #103292
• Revert "gh-103224: Use the realpath of the Python executable in test_venv (GH-103243)" #108667
• [3.12] gh-103224: Resolve paths properly in test_sysconfig (GH-103292) #115100
• [3.11] gh-103224: Resolve paths properly in test_sysconfig (GH-103292) #115101

[dtrodrigues]

What if you do python.exe -m test or python.exe -m test test_venv --verbose?

[artemmukhin]

@dtrodrigues Both python.exe -m test and python.exe -m test test_venv --verbose finish without failures. test_sysconfig passes only when I run ./python.exe and not the symlink python.exe.

I have opened a PR regarding test_venv.

[artemmukhin]

Regarding test_sysconfig, I found out that replacing sys.prefix with sys.base_prefix makes test_sysconfig passes with both ./python.exe and python.exe symlink.

Although I'm not entirely sure if this is the correct way to fix the test. So before opening the next PR regarding test_sysconfig, I would like to confirm this is the appropriate solution.

@hroncok Could you please take a look if you have a chance? Here is a patch:

diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index b6dbf3d52c..a95e7f4367 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -151,7 +151,7 @@ def test_posix_venv_scheme(self):

         # Resolve the paths in prefix
         binpath = os.path.join(sys.prefix, binpath)
-        incpath = os.path.join(sys.prefix, incpath)
+        incpath = os.path.join(sys.base_prefix, incpath)
         libpath = os.path.join(sys.prefix, libpath)

         self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv'))
@@ -171,7 +171,7 @@ def test_nt_venv_scheme(self):

         # Resolve the paths in prefix
         binpath = os.path.join(sys.prefix, binpath)
-        incpath = os.path.join(sys.prefix, incpath)
+        incpath = os.path.join(sys.base_prefix, incpath)
         libpath = os.path.join(sys.prefix, libpath)

         self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv'))

[hroncok]

Will do! It has been a while since I wrote that, but I'll try to figure it out.

[hroncok]

The include path is indeed defined via installed_base and the default for insatlled_base is sys.base_prefix, so the diff makes the test pass but arguably makes the test more fragile for potential future posix_venv changes.

I suggest instead of "Resolve the paths in prefix" we resolve the paths in "venv/" and then we set vars to a dict similar to the one in

cpython/Lib/venv/__init__.py

Lines 97 to 102 in f513d5c

	vars = {
	'base': env_dir,
	'platbase': env_dir,
	'installed_base': env_dir,
	'installed_platbase': env_dir,
	}

when calling get_path.

Something like this (untested):

diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index b6dbf3d52cb..1137c2032b9 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -149,17 +149,21 @@ def test_posix_venv_scheme(self):
                                'python%d.%d' % sys.version_info[:2],
                                'site-packages')
 
-        # Resolve the paths in prefix
-        binpath = os.path.join(sys.prefix, binpath)
-        incpath = os.path.join(sys.prefix, incpath)
-        libpath = os.path.join(sys.prefix, libpath)
+        # Resolve the paths in an imaginary venv/ directory
+        binpath = os.path.join('venv', binpath)
+        incpath = os.path.join('venv', incpath)
+        libpath = os.path.join('venv', libpath)
 
-        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv'))
-        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='posix_venv'))
+        # Mimic the venv module, set all bases to the venv directory
+        bases = ('base', 'platbase', 'installed_base', 'installed_platbase')
+        vars = {base: 'venv' for base in bases}
+
+        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv', vars=vars))
+        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='posix_venv', vars=vars))
 
         # The include directory on POSIX isn't exactly the same as before,
         # but it is "within"
-        sysconfig_includedir = sysconfig.get_path('include', scheme='posix_venv')
+        sysconfig_includedir = sysconfig.get_path('include', scheme='posix_venv', vars=vars)
         self.assertTrue(sysconfig_includedir.startswith(incpath + os.sep))
 
     def test_nt_venv_scheme(self):
@@ -169,14 +173,19 @@ def test_nt_venv_scheme(self):
         incpath = 'Include'
         libpath = os.path.join('Lib', 'site-packages')
 
-        # Resolve the paths in prefix
-        binpath = os.path.join(sys.prefix, binpath)
-        incpath = os.path.join(sys.prefix, incpath)
-        libpath = os.path.join(sys.prefix, libpath)
+        # Resolve the paths in an imaginary venv\ directory
+        venv = 'venv'
+        binpath = os.path.join(venv, binpath)
+        incpath = os.path.join(venv, incpath)
+        libpath = os.path.join(venv, libpath)
+
+        # Mimic the venv module, set all bases to the venv directory
+        bases = ('base', 'platbase', 'installed_base', 'installed_platbase')
+        vars = {base: 'venv' for base in bases}
 
-        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv'))
-        self.assertEqual(incpath, sysconfig.get_path('include', scheme='nt_venv'))
-        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='nt_venv'))
+        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv', vars=vars))
+        self.assertEqual(incpath, sysconfig.get_path('include', scheme='nt_venv', vars=vars))
+        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='nt_venv', vars=vars))
 
     def test_venv_scheme(self):
         if sys.platform == 'win32':

I think that this should have been done initially and my test was buggy.

[artemmukhin]

@hroncok Thank you! Your patch fixes the tests for me, so I have opened a PR #103292

[zooba]

This change causes tests to break when configured and run through a prefix:

./configure --with-openssl=/usr/local/opt/openssl \
            --prefix=$(Py_TargetDir) \
            --enable-shared \
            --enable-universalsdk --with-universal-archs=intel-64
make -j4 install
make buildbottest TESTOPTS="-j4 -uall,-cpu --tempdir=$(Py_IntDir)/tests"

macOS Logs:

======================================================================
ERROR: test_zippath_from_non_installed_posix (test.test_venv.BasicTest.test_zippath_from_non_installed_posix)
Test that when create venv from non-installed python, the zip path
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/1/a/layout/lib/python3.13/test/test_venv.py", line 612, in test_zippath_from_non_installed_posix
    subprocess.check_call(cmd, env=child_env)
  File "/Users/runner/work/1/a/layout/lib/python3.13/subprocess.py", line 408, in check_call
    retcode = call(*popenargs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/1/a/layout/lib/python3.13/subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/1/a/layout/lib/python3.13/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/runner/work/1/a/layout/lib/python3.13/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_ceck5p2j/tmpv23r1zhj/bin/python3'

======================================================================
FAIL: test_upgrade_dependencies (test.test_venv.BasicTest.test_upgrade_dependencies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/1/a/layout/lib/python3.13/test/test_venv.py", line 236, in test_upgrade_dependencies
    builder.upgrade_dependencies(fake_context)
  File "/Users/runner/work/1/a/layout/lib/python3.13/venv/__init__.py", line 459, in upgrade_dependencies
    self._call_new_python(context, '-m', 'pip', 'install', '--upgrade',
  File "/Users/runner/work/1/a/layout/lib/python3.13/venv/__init__.py", line 355, in _call_new_python
    subprocess.check_output(args, **kwargs)
  File "/Users/runner/work/1/a/layout/lib/python3.13/test/test_venv.py", line 222, in pip_cmd_checker
    self.assertEqual(
AssertionError: Lists differ: ['/va[76 chars]ct/bin/python3', '-m', 'pip', 'install', '--upgrade', 'pip'] != ['/va[76 chars]ct/bin/python3.13', '-m', 'pip', 'install', '--upgrade', 'pip']

First differing element 0:
'/var[30 chars]sl6xvm0000gn/T/test_python_ceck5p2j/tmp37gcakct/bin/python3'
'/var[30 chars]sl6xvm0000gn/T/test_python_ceck5p2j/tmp37gcakct/bin/python3.13'

- ['/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_ceck5p2j/tmp37gcakct/bin/python3',
+ ['/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/test_python_ceck5p2j/tmp37gcakct/bin/python3.13',
?                                                                                                +++

   '-m',
   'pip',
   'install',
   '--upgrade',
   'pip']

I get the same failure (slightly different paths) on Ubuntu.

Any other thoughts on how to solve the original issue?

[hroncok]

Even with #103292 ?

[zooba]

I'm just running against what's in main, so I don't know yet :)

I'll patch it into my repo and have a go

[zooba]

But it looks like that only affects test_sysconfig, and so won't impact the changes made by #103243 ?

[hroncok]

No idea. The change in PR I linked should fix problems in the test when custom prefix is set. It's long time since it was proposed and I lost all the context.

[zooba]

Well, it doesn't give me any more failures at least.

I think the problem is that make buildbottest launches through a different symlink after make install than the usual test suite? Or it's doing realpath in a place where venv does not (which is obviously true, because this only started failing when the extra realpath was added, but that doesn't mean that simply reverting is the right fix).

Clearly we just need to make the test robust enough for both scenarios, as both are legitimate. I'm not sure what the original problem being solved is though, so the best I can do is try to revert the change.

[zooba]

FWIW, I fixed this for myself by running tests through python3.x rather than through python3.

I still think we should be able to handle both, but at least it's probably just a test issue.

[serhiy-storchaka]

#103243 broke some builtbots.
https://buildbot.python.org/all/#/builders/363/builds/1147/steps/7/logs/stdio
https://buildbot.python.org/all/#/builders/292/builds/1142/steps/7/logs/stdio
https://buildbot.python.org/all/#/builders/356/builds/1153/steps/7/logs/stdio
https://buildbot.python.org/all/#/builders/465/builds/1204/steps/7/logs/stdio

======================================================================
ERROR: test_zippath_from_non_installed_posix (test.test_venv.BasicTest.test_zippath_from_non_installed_posix)
Test that when create venv from non-installed python, the zip path
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 612, in test_zippath_from_non_installed_posix
    subprocess.check_call(cmd, env=child_env)
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/subprocess.py", line 408, in check_call
    retcode = call(*popenargs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp_6mtreoe/bin/python3'
======================================================================
FAIL: test_upgrade_dependencies (test.test_venv.BasicTest.test_upgrade_dependencies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 236, in test_upgrade_dependencies
    builder.upgrade_dependencies(fake_context)
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/venv/__init__.py", line 459, in upgrade_dependencies
    self._call_new_python(context, '-m', 'pip', 'install', '--upgrade',
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/venv/__init__.py", line 355, in _call_new_python
    subprocess.check_output(args, **kwargs)
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 222, in pip_cmd_checker
    self.assertEqual(
AssertionError: Lists differ: ['/tmp/tmp77w4qegx/bin/python3', '-m', 'pip', 'install', '--upgrade', 'pip'] != ['/tmp/tmp77w4qegx/bin/python3.13', '-m', 'pip', 'install', '--upgrade', 'pip']
First differing element 0:
'/tmp/tmp77w4qegx/bin/python3'
'/tmp/tmp77w4qegx/bin/python3.13'
- ['/tmp/tmp77w4qegx/bin/python3', '-m', 'pip', 'install', '--upgrade', 'pip']
+ ['/tmp/tmp77w4qegx/bin/python3.13', '-m', 'pip', 'install', '--upgrade', 'pip']
?                               +++
----------------------------------------------------------------------

It can be easily reproduced by creating a symlink to Python executable and running tests via the symlink.

ln -s python pythonx
./pythonx -m test -vuall test_venv

Reverting #103243 fixes this issue. I actually first wrote a fix myself, but it turned out to be equivalent to the reversion of #103243.

[vsajip]

I've approved the reversion - @artemmukhin perhaps the change needed to be more narrowly applied? If you want to work up a PR we can test with the build-bots first before finalizing it.

[serhiy-storchaka]

I have no macOS. Can the original issue be reproduced on non-macOS? What exactly the failure report, which lines fail and with what output?