pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8 #8272

glyph · 2020-05-19T20:36:19Z

Environment

pip version: 20.1, 20.1.1
Python version: 3.8.2
OS: macOS, linux

Description
pip install -e fails with a traceback if the typing module is installed from PyPI.

Expected behavior
It should work, like it used to.

How to Reproduce

python3.8 -m venv foo
./foo/bin/pip install typing
./foo/bin/pip install -e ~/src/twisted

Output

  ERROR: Command errored out with exit status 1:
   command: /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/bin/python /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/z9/w23lcw3x70z1fqpcwjjbvkvc0000gp/T/pip-build-env-f7o2d80a/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i http://127.0.0.1:3141/root/plus/+simple/ -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (44 lines):
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/__main__.py", line 26, in <module>
      sys.exit(_main())
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 73, in main
      command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
      module = importlib.import_module(module_path)
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 783, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 24, in <module>
      from pip._internal.cli.req_command import RequirementCommand, with_cleanup
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
      from pip._internal.index.package_finder import PackageFinder
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 21, in <module>
      from pip._internal.index.collector import parse_links
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 14, in <module>
      from pip._vendor import html5lib, requests
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py", line 114, in <module>
      from . import utils
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
      from . import certs
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
      from pip._vendor.certifi import where
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
      from .core import contents, where
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
      from importlib.resources import read_text
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/resources.py", line 11, in <module>
      from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/typing.py", line 1357, in <module>
      class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/typing.py", line 1005, in __new__
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/bin/python /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/z9/w23lcw3x70z1fqpcwjjbvkvc0000gp/T/pip-build-env-f7o2d80a/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i http://127.0.0.1:3141/root/plus/+simple/ -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

The text was updated successfully, but these errors were encountered:

glyph · 2020-05-19T21:25:03Z

I think this might just be a duplicate of #8214 ?

pfmoore · 2020-05-19T21:32:36Z

It is indeed.

It should work, like it used to.

When you say "it used to", when was the last time it worked? My feeling is that it would not have worked since we introduced build isolation. Does that match your experience?

glyph · 2020-05-20T05:50:34Z

It worked in every version prior to 20.1, so 20.0.2.

pfmoore · 2020-05-20T07:15:17Z

Thanks, that's useful to know.

pradyunsg · 2020-05-20T07:26:50Z

Gah, this is likely a regression, since nearly all of pip's code avoids importing typing as much as possible. I'm curious how this slipped through, but we can look into this separately.

Looks like another bugfix release is needed. :(

pfmoore · 2020-05-20T07:49:10Z

It looks like #5841 introduced this, triggering the issue with how we set sys.path in __main__.

pfmoore · 2020-05-20T07:55:05Z

I'm now inclined to accept #8213 as a fix for this. It's a bit of a hack, but so's all of the code in this area, and now that we understand what's going on, and have confirmation that it's affecting things more widely than the original report suggested, it seems better to apply that fix and worry later about cleaning up build_env a bit better.

uranusjr · 2020-05-20T11:21:44Z

I think it’d be better to just revert #5841 than accepting #8213. They would both break the usage #5841 tried to fix, but at least reversion is unlikely to introduce more problems.

pfmoore · 2020-05-20T12:08:16Z

Ah sorry, you're right. I was too quick in my comment - indeed #8213 breaks #5841, I was reading the conditional wrongly.

I still think we need to understand the problem better, and I don't think the issue is urgent enough to need another bugfix release (all of the reproducers I've seen involve installing a backport module in a Python release that doesn't need it - so "don't do that" seems like a sufficient workaround for the immediate term).

One thing that bothers me - although I haven't tested anything myself - is that this is reported as working in 20.0.2, which is odd as #5841 was included from 19.0, and the sys.path code has been there since forever. So what changed after 20.0.2 that affected this?

pradyunsg · 2020-05-20T12:32:05Z

I don't think the issue is urgent enough to need another bugfix release (all of the reproducers I've seen involve installing a backport module in a Python release that doesn't need it - so "don't do that" seems like a sufficient workaround for the immediate term).

Yea, that makes sense to me! :)

tony · 2020-06-21T17:32:58Z

Possibly related

mypy-extensions https://pypi.org/project/typing-extensions/#history

3.7.4.2 on Apr 2nd, 2020

python/typing@f254d69#diff-85eeb059061bb009c803d065dce510ec

https://github.com/python/typing/commits/3.7.4.1/typing_extensions/setup.py

uranusjr · 2020-07-24T10:03:33Z

Aggregating the conversation into #8214, which contains more context (with PRs linked to it).

- #11 - pypa/pip#8214 - pypa/pip#8214

To prevent pypa/pip#8272 downstream.

This prevents shadowing the version of the module that comes with the stdlib since Python 3.5. Quoting from the package pypi site (https://pypi.org/project/typing/): "For package maintainers, it is preferred to use typing;python_version<"3.5" if your package requires it to support earlier Python versions. This will avoid shadowing the stdlib typing module when your package is installed via pip install -t . on Python 3.5 or later." Incidentally, this fixes some bug caused by the two versions of typing coexisting that arises in some versions of Python 3.7, see e.g.: pypa/pip#8272 https://travis-ci.community/t/python-3-7-and-3-8-builds-are-failing-attributeerror-type-object-callable-has-no-attribute-abc-registry/10900

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label May 19, 2020

pquentin mentioned this issue May 25, 2020

Stop installing typing on Python 3.5+ mbarkhau/bumpver#148

Merged

moriyoshi mentioned this issue Jun 8, 2020

AttributeError: type object 'Callable' has no attribute '_abc_registry' ethereum/eth-abi#131

Closed

tony mentioned this issue Jun 21, 2020

type object 'Callable' has no attribute '_abc_registry' xmlsec/python-xmlsec#139

Closed

atugushev mentioned this issue Jul 3, 2020

Alembic 1.4.2 breaks pip-compile 5.2.1 jazzband/pip-tools#1171

Closed

This was referenced Jul 7, 2020

CI failure because of pip basnijholt/aiokef#11

Closed

_abc_registry errors with py3.8 python/typing#721

Closed

McSinyx mentioned this issue Jul 24, 2020

ERROR: Command errored out with exit status 1 #8618

Closed

uranusjr added the type: bug A confirmed bug or unintended behavior label Jul 24, 2020

triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Jul 24, 2020

uranusjr closed this as completed Jul 24, 2020

uranusjr added resolution: duplicate Duplicate of an existing issue/PR and removed type: bug A confirmed bug or unintended behavior labels Jul 24, 2020

pfmoore mentioned this issue Jul 24, 2020

With Enum34 installed on python3.6+, pip picks up enum34 instead of the stdlib enum package #8214

Closed

The-Compiler mentioned this issue Aug 4, 2020

Properly remove log_print pytest-dev/pytest#7614

Merged

bluetech mentioned this issue Aug 4, 2020

CI fails with "AttributeError: type object 'Callable' has no attribute '_abc_registry'" pytest-dev/pytest#7616

Closed

reaperhulk mentioned this issue Aug 4, 2020

typing installed unconditionally on all Pythons when installed from wheel python-hyper/hyperlink#133

Closed

chocoelho mentioned this issue Aug 22, 2020

[BUG] Can't install prospector 1.3.0 on python 3.7 (buster) prospector-dev/prospector#391

Closed

cdb39 mentioned this issue Sep 2, 2020

Only install typing on python versions less than 3.5 palantir/conjure-python#371

Closed

basnijholt added a commit to basnijholt/aiokef that referenced this issue Sep 16, 2020

uninstall tpying, pypa/pip#8214, pypa/pip#8272

5753c8a

- #11 - pypa/pip#8214 - pypa/pip#8214

basnijholt added a commit to basnijholt/aiokef that referenced this issue Sep 16, 2020

uninstall tpying, pypa/pip#8214, pypa/pip#8272

629e68e

- #11 - pypa/pip#8214 - pypa/pip#8214

basnijholt added a commit to basnijholt/async-timeout that referenced this issue Sep 16, 2020

Only install typing_extensions for Python 3.5 and 3.6

078d521

To prevent pypa/pip#8272 downstream.

basnijholt mentioned this issue Sep 16, 2020

Only install typing_extensions for Python 3.5 and 3.6 aio-libs/async-timeout#174

Closed

4 tasks

awilkins mentioned this issue Oct 1, 2020

typing breaks local package installs python-poetry/poetry#3020

Closed

3 tasks

AKAMEDIASYSTEM mentioned this issue Oct 11, 2020

Anyleaf ph, install dependencies kizniche/Mycodo#863

Closed

Harmon758 mentioned this issue Dec 17, 2020

Error upgrading discord.py to 1.5.1 Rapptz/discord.py#6170

Closed

gfrances mentioned this issue May 12, 2021

Install typing module only for python<3.5 thiagopbueno/pyrddl#12

Merged

github-actions bot locked as resolved and limited conversation to collaborators Oct 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8 #8272

pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8 #8272

glyph commented May 19, 2020

glyph commented May 19, 2020

pfmoore commented May 19, 2020

glyph commented May 20, 2020

pfmoore commented May 20, 2020

pradyunsg commented May 20, 2020

pfmoore commented May 20, 2020

pfmoore commented May 20, 2020

uranusjr commented May 20, 2020 •

edited

Loading

pfmoore commented May 20, 2020

pradyunsg commented May 20, 2020

tony commented Jun 21, 2020

uranusjr commented Jul 24, 2020

pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8 #8272

pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8 #8272

Comments

glyph commented May 19, 2020

glyph commented May 19, 2020

pfmoore commented May 19, 2020

glyph commented May 20, 2020

pfmoore commented May 20, 2020

pradyunsg commented May 20, 2020

pfmoore commented May 20, 2020

pfmoore commented May 20, 2020

uranusjr commented May 20, 2020 • edited Loading

pfmoore commented May 20, 2020

pradyunsg commented May 20, 2020

tony commented Jun 21, 2020

uranusjr commented Jul 24, 2020

uranusjr commented May 20, 2020 •

edited

Loading