Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] new release (57) breaks piptools pytest #2687

Closed
1 task done
nicoa opened this issue May 28, 2021 · 21 comments
Closed
1 task done

[BUG] new release (57) breaks piptools pytest #2687

nicoa opened this issue May 28, 2021 · 21 comments
Labels

Comments

@nicoa
Copy link

nicoa commented May 28, 2021

First things first: please let me know if there is any information missing - It's kind of hard to me to trace this down to the actual bug or to an minimum example of what is failing unfortunately.

setuptools version

setuptools==57.0.0

Python version

3.6,3.7,3.8,3.9

OS

Ubuntu, Windows, MacOS

Additional environment information

Hey, please see this issue where it was discovered: jazzband/pip-tools#1403

First, it was wrongly attributed to be caused by pip version deprecation and feature flag, but it seems to be due to new setuptools (details in description). So currently all CI runners are broken there in that project.

Description

Unfortunately, I'm not exactly aware of what happens behind the scenes on piptools testing suite. I'd like to share the code I could reproduce locally so far by running some of the tests manually in the fields below.

Expected behavior

There shouldn't be anything failing because it worked on the old version before.

How to Reproduce

  1. in piptools package open tests/test_cli_compile.py and copy the imports from above and the code from the function test_realistic_complex_sub_dependencies, done in the following in detail:
  2. navigate to tests folder and run the following:
import os
import subprocess
import sys
from textwrap import dedent
from unittest import mock
import pytest
from pip._internal.utils.urls import path_to_url
from piptools.scripts.compile import cli
from constants import MINIMAL_WHEELS_PATH, PACKAGES_PATH
from click.testing import CliRunner
  1. create runner = CliRunner(mix_stderr=False)
  2. run
    wheels_dir = "wheels"
    # make a temporary wheel of a fake package
    subprocess.run(
        [
            "pip",
            "wheel",
            "--no-deps",
            "-w",
            wheels_dir,
            os.path.join(PACKAGES_PATH, "fake_with_deps", "."),
        ],
        check=True,
    )
    with open("requirements.in", "w") as req_in:
        req_in.write("fake_with_deps")  # require fake package

    out = runner.invoke(cli, ["-n", "--rebuild", "-f", wheels_dir])

which fails.

Output

expected

In [39]: out.stderr
Out[39]: '#\n# This file is autogenerated by pip-compile\n# To update, run:\n#\n#    pip-compile --find-links=wheels\n#\n--find-links wheels\n\ncolorama==0.3.9\n    # via fake-with-deps\ncornice==1.0.0\n    # via fake-with-deps\nenum34==1.1.6\n    # via fake-with-deps\nfake-with-deps==0.1\n    # via -r requirements.in\nfutures==3.0.5\n    # via fake-with-deps\ngreenlet==1.1.0\n    # via sqlalchemy\nipaddress==1.0.23\n    # via fake-with-deps\njsonschema==2.6.0\n    # via fake-with-deps\npastedeploy==2.1.1\n    # via pyramid\npyramid==1.5.8\n    # via\n    #   cornice\n    #   fake-with-deps\npython-dateutil==2.4.2\n    # via fake-with-deps\npython-memcached==1.59\n    # via fake-with-deps\npyzmq==14.7.0\n    # via fake-with-deps\nrepoze.lru==0.7\n    # via pyramid\nsimplejson==3.17.2\n    # via\n    #   cornice\n    #   fake-with-deps\nsix==1.8.0\n    # via\n    #   fake-with-deps\n    #   python-dateutil\n    #   python-memcached\nsqlalchemy==1.4.15\n    # via fake-with-deps\ntranslationstring==1.4\n    # via pyramid\nvenusian==3.0.0\n    # via pyramid\nwebob==1.8.7\n    # via pyramid\nxmltodict==0.11.0\n    # via fake-with-deps\nzope.deprecation==4.4.0\n    # via pyramid\nzope.interface==5.4.0\n    # via pyramid\n\n# The following packages are considered to be unsafe in a requirements file:\n# setuptools\nDry-run, so nothing updated.\n'

actual (failing) output

      ERROR: Command errored out with exit status 1:
       command: /Users/nalbers/git/pip-tools/thisvirtualenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"'; __file__='"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-pip-egg-info-nkzz3eb8
           cwd: /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/
      Complete output (29 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/__init__.py", line 18, in <module>
          from setuptools.dist import Distribution
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/_virtualenv.py", line 89, in exec_module
          old(module)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/dist.py", line 32, in <module>
          from setuptools.extern.more_itertools import unique_everseen
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 52, in create_module
          return self.load_module(spec.name)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 37, in load_module
          __import__(extant)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
          from .more import *  # noqa
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
          from concurrent.futures import ThreadPoolExecutor
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/__init__.py", line 8, in <module>
          from concurrent.futures._base import (FIRST_COMPLETED,
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/_base.py", line 357
          raise type(self._exception), self._exception, self._traceback
                                     ^
      SyntaxError: invalid syntax
      ----------------------------------------

Code of Conduct

  • I agree to follow the PSF Code of Conduct
@nicoa nicoa added Needs Triage Issues that need to be evaluated for severity and status. bug labels May 28, 2021
@atugushev
Copy link

atugushev commented Oct 8, 2021

git bisect detects 3544de7 commit. Minimal reproducer:

# setup.py
from setuptools import setup

setup(
    name="foo",
    version=0.1,
    install_requires=[
        "futures<3.1,>=3.0.3",
    ],
)
$ pip install -e . --no-cache --force-reinstall

@atugushev
Copy link

atugushev commented Oct 8, 2021

Ah, finally got it. Since 57 version setuptools uses more-itertools package, which uses/imports concurrent.futures module, which happens to be broken after pip install futures, since futures backports concurrent.futures to Python 2.

File "/usr/lib/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
  from concurrent.futures import ThreadPoolExecutor

@jaraco Looks like it's more like a bug on https://github.com/agronholm/pythonfutures side. It probalby should use python_requires<3 in setup.py (agronholm/pythonfutures#95).

@jaraco
Copy link
Member

jaraco commented Oct 8, 2021

Thanks for the investigation. Sounds like there's not much for Setuptools to do here. Feel free to comment if I'm mistaken and we can re-open.

@jaraco jaraco closed this as completed Oct 8, 2021
@jaraco jaraco added invalid and removed bug Needs Triage Issues that need to be evaluated for severity and status. labels Oct 8, 2021
@agronholm
Copy link
Contributor

As pointed out on my side, futures already declares a maximum Python version.

@jaraco
Copy link
Member

jaraco commented Oct 8, 2021

It sounds as if something in the build chain is not honoring the python-requires then.

@agronholm
Copy link
Contributor

There are earlier releases that don't have this declaration, maybe that's why. @nicoa did you check which version was installed?

Maybe I should yank all those obsolete releases.

@atugushev
Copy link

It tries to install futures<3.1,>=3.0.3, but python_requires introduced in 3.2.0. It works fine with the latest futures version.

@agronholm
Copy link
Contributor

It tries to install futures<3.1,>=3.0.3, but python_requires introduced in 3.2.0. It works fine with the latest futures version.

So what exactly has this dependency? pip-tools doesn't, setuptools doesn't, more-itertools doesn't.

@agronholm
Copy link
Contributor

I made a post-release that retroactively adds python-requires, and yanked 3.1.1. The earlier releases should be syntax compatible at least.

@PatTheMav
Copy link

I still get the reported issue with Python 3.9 and a package that specifies futures>=3.1.0 in its requirements.txt - with setuptools v57+ installed, all versions down to futures v2.2.0 are iterated through (each one failing with SyntaxError: invalid syntax), which also leads to an older version of the original package being installed.

I couldn't figure out how >= 3.1.0 which should yield at least the new 3.1.1 release or any more recent release that introduced the python-requires statement.

The only way to fix this for me was to downgrade setuptools to a version older than 57 - futures will not be installed and the correct version of the original package is installed.

@agronholm
Copy link
Contributor

What's the simplest way to repro this?

@PatTheMav
Copy link

Try pip install modoboa-dmarc with pip 21.3 and setup tools 58.2.0, I run this in a Python 3.9 virtualenv.

@agronholm
Copy link
Contributor

Ok so...if I was to yank 3.1.0, what would happen? The installation would fail because there is no matching release.

@agronholm
Copy link
Contributor

What also greatly confuses me is that 3rd party packages aren't supposed to override stdlib imports. Why is that happening here?

@PatTheMav
Copy link

PatTheMav commented Oct 12, 2021

What also greatly confuses me is that 3rd party packages aren't supposed to override stdlib imports. Why is that happening here?

From what I could tell that seems like the main issue - if I'm not mistaken, specifying >=3.1.0 will resolve to 3.1.0 as the only viable candidate under Python 3. Under normal circumstances import concurrent.futures seems to work fine, even with futures being installed (probably due to stdlib imports taking precedence as you mentioned before), so somehow this doesn't apply to setuptools/more_itertools.

EDIT: Turns out that the current working directory takes precedence even over the standard library. So if I create a module called math.py and run import math in an interpreter in the same directory, my own module is imported.

What seems to happen here is that setuptools runs in the temporary install directory (pip-resolver-x2nbtsp4/futures/ in the example above) which contains the actual local path "concurrent/futures". The import statement of more_itertools will thus pick up the futures implementation of futures itself and lead to the SyntaxError.

@jaraco
Copy link
Member

jaraco commented Oct 19, 2021

I can more readily replicate the issue by simply attempting to install futures with the latest setuptools present:

~ $ pip-run -q setuptools -- -m pip-run futures==3.1
Collecting futures==3.1
  Using cached futures-3.1.0.tar.gz (25 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-pip-egg-info-nmdijp9m
       cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/
  Complete output (24 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/__init__.py", line 18, in <module>
      from setuptools.dist import Distribution
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/dist.py", line 32, in <module>
      from setuptools.extern.more_itertools import unique_everseen
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 571, in module_from_spec
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/extern/__init__.py", line 52, in create_module
      return self.load_module(spec.name)
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/extern/__init__.py", line 37, in load_module
      __import__(extant)
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
      from .more import *  # noqa
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
      from concurrent.futures import ThreadPoolExecutor
    File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/concurrent/futures/__init__.py", line 8, in <module>
      from concurrent.futures._base import (FIRST_COMPLETED,
    File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/concurrent/futures/_base.py", line 357
      raise type(self._exception), self._exception, self._traceback
                                 ^
  SyntaxError: invalid syntax
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/25/83/113da7c7832a4efa80ec7fd5d25b2ea431d9a847cd78f1fa98ddd99d591a/futures-3.1.0.tar.gz#sha256=90a8c71762d53e4ebdb4f7893460761f4ce2f076df08bb4934f39e90464eb889 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement futures==3.1 (from versions: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1)
ERROR: No matching distribution found for futures==3.1
...

@jaraco
Copy link
Member

jaraco commented Oct 19, 2021

I note that employing --use-pep517 for the install of futures works around the issue:

~ $ pip-run -q setuptools -- -m pip-run --use-pep517 futures==3.1 -- -c "print('worked')"
Collecting futures==3.1
  Using cached futures-3.1.0-py3-none-any.whl
Installing collected packages: futures
Successfully installed futures-3.1.0
worked

Here's what I think is happening:

  • The futures distribution does not supply a pyproject.toml, so builds fallback to the legacy installer.
  • The legacy installer retains the legacy behavior that during install of a package, that package's root is added to sys.path.
  • During the install of futures, more_itertools gets concurrent.futures from the local build of futures.

Next, I'll attempt to replicate the failure without pip by simulating what pip is doing.

@jaraco
Copy link
Member

jaraco commented Oct 19, 2021

Indeed, I can replicate the issue without pip thus:

draft $ http https://files.pythonhosted.org/packages/25/83/113da7c7832a4efa80ec7fd5d25b2ea431d9a847cd78f1fa98ddd99d591a/futures-3.1.0.tar.gz | tar xz
draft $ cd futures-3.1.0/
futures-3.1.0 $ pip-run -q setuptools -- setup.py egg_info
/Users/jaraco/draft/futures-3.1.0/setup.py:6: UserWarning: This backport is meant only for Python 2.
Python 3 users do not need it, as the concurrent.futures package is available in the standard library.
  warn('This backport is meant only for Python 2.\n'
Traceback (most recent call last):
  File "/Users/jaraco/draft/futures-3.1.0/setup.py", line 12, in <module>
    from setuptools import setup
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/dist.py", line 32, in <module>
    from setuptools.extern.more_itertools import unique_everseen
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 52, in create_module
    return self.load_module(spec.name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 37, in load_module
    __import__(extant)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
    from .more import *  # noqa
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
    from concurrent.futures import ThreadPoolExecutor
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/_base.py", line 357
    raise type(self._exception), self._exception, self._traceback
                               ^
SyntaxError: invalid syntax
futures-3.1.0 $ python -c "import concurrent; print(concurrent.__file__)"
/Users/jaraco/draft/futures-3.1.0/concurrent/__init__.py

@jaraco
Copy link
Member

jaraco commented Oct 19, 2021

So if futures is installed, it won't take precedence to the stdlib (site-packages occurs after builtins), but if futures is in ., it will take precedence.

@jaraco
Copy link
Member

jaraco commented Oct 19, 2021

And the only reason this issue was not an issue before was because nothing in Setuptools attempted to import concurrent.futures until more_itertools was added, so the failing code path wasn't triggered. Based on my reading of the package, it should never be installed on Python 3, so the packages that are depending on it should stop doing so for Python 3.

@PatTheMav
Copy link

Yep exactly, it's the perfect storm of setuptools using a dependency that imports concurrent.futures since v57 and root imports having higher priority than stdlib imports (which "protected" Py3 against this incompatible package so far).

Only installing the futures package itself could lead to this issue, as it's the only one containing an actual concurrent.futures module (or more precisely, any package that has such a module with code that isn't Py3 compatible) available at the project root.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants
@agronholm @jaraco @nicoa @atugushev @PatTheMav and others