Module installation fails due to missing dependency

[nadav-ravid]

redis-py/redis/connection.py

Line 1 in 039488d

from packaging.version import Version

the deprecated distutils was replaced with the packaging module as part of release v4.0.0b1
packaging is not a builtin python module but was not added to setup.py as a dependency which causes applications that require redis-py to fail if packaging isn't already installed on the machine.
the packaging module should probably be added as a dependency in setup.py to resolve this

[chayim]

Thanks for pointing this out. This definitely snuck in, as it's a package that's default distutils-ed on several OSes (including all 3 of mine), so I didn't notice.

[chayim]

Fixed in #1626 . Closing.

[nadav-ravid]

Happy to help :)

However the merged fix seems counter productive since distutils is considered deprecated and is scheduled for removal from the python standard library in python 3.12 (which i think was the reason for the original change)

[hartwork]

Please re-open, pull request #1626 did not fix this issue (and is indeed a step backwards), pip install . is still broken on master. To reproduce:

# cd "$(mktemp -d)"
# git clone --depth 1 https://github.com/andymccurdy/redis-py
# cd redis-py/
# virtualenv --python=python3.9 venv
# source venv/bin/activate
# pip install .
Processing /tmp/tmp.YtqtDgr1Ry/redis-py
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /tmp/tmp.YtqtDgr1Ry/redis-py/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/tmp.YtqtDgr1Ry/redis-py/setup.py'"'"'; __file__='"'"'/tmp/tmp.YtqtDgr1Ry/redis-py/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-9ygcp0w4
       cwd: /tmp/tmp.YtqtDgr1Ry/redis-py/
  Complete output (11 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/tmp.YtqtDgr1Ry/redis-py/setup.py", line 4, in <module>
      import redis
    File "/tmp/tmp.YtqtDgr1Ry/redis-py/redis/__init__.py", line 1, in <module>
      from redis.client import Redis, StrictRedis
    File "/tmp/tmp.YtqtDgr1Ry/redis-py/redis/client.py", line 15, in <module>
      from redis.connection import ConnectionPool, SSLConnection, UnixDomainSocketConnection
    File "/tmp/tmp.YtqtDgr1Ry/redis-py/redis/connection.py", line 13, in <module>
      from packaging.version import Version
  ModuleNotFoundError: No module named 'packaging'
  ----------------------------------------
WARNING: Discarding file:///tmp/tmp.YtqtDgr1Ry/redis-py. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I believe the issue is that packaging is not in setup_requires (aka "build dependencies") in setup.py but only in install_requires (aka runtime dependencies).

The == pinning in setup.py also blocks users from updating, so I believe we need something like this here, for a full fix:

# git --no-pager diff
diff --git a/setup.py b/setup.py
index 58d753f..b9c2e3e 100644
--- a/setup.py
+++ b/setup.py
@@ -26,9 +26,12 @@ setup(
     author="Redis Inc.",
     author_email="oss@redis.com",
     python_requires=">=3.6",
+    setup_requires=[
+        "packaging>=21.3",
+    ],
     install_requires=[
-        "deprecated==1.2.3",
-        "packaging==21.3",
+        "deprecated>=1.2.3",
+        "packaging>=21.3",
     ],
     classifiers=[
         "Development Status :: 5 - Production/Stable",

I will try turn that into a pull request in a minute.

[ashwani99]

Hi @hartwork I have been following this issue and trying to reproduce this issue while I'm on #1781

Strangely, I am able to reproduce the issue even after #1780 is merged. Here are the steps I did:

$ cd "$(mktemp -d)"
$ git clone --depth 1 https://github.com/redis/redis-py
$ cd redis-py
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install .
Processing /tmp/tmp.acCdF6XYQC/redis-py
    ERROR: Command errored out with exit status 1:
     command: /tmp/tmp.acCdF6XYQC/redis-py/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nvv2qnn6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nvv2qnn6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-nvv2qnn6/pip-egg-info
         cwd: /tmp/pip-req-build-nvv2qnn6/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-nvv2qnn6/setup.py", line 4, in <module>
        import redis
      File "/tmp/pip-req-build-nvv2qnn6/redis/__init__.py", line 1, in <module>
        from redis.client import Redis, StrictRedis
      File "/tmp/pip-req-build-nvv2qnn6/redis/client.py", line 15, in <module>
        from redis.connection import ConnectionPool, SSLConnection, UnixDomainSocketConnection
      File "/tmp/pip-req-build-nvv2qnn6/redis/connection.py", line 13, in <module>
        from packaging.version import Version
    ModuleNotFoundError: No module named 'packaging'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Am I missing something here? I am new to the internals of setuptools and how pip handles install_requires and setup_requires, would appreciate your help :)

[hartwork]

@ashwani99 that's a limitation of setup_requires to my understanding, see #1781 (comment) and https://pip.pypa.io/en/stable/reference/build-system/#controlling-setup-requires for more details please. Does that help?

[ashwani99]

@hartwork Thanks, that helped. From setuptools docs https://setuptools.pypa.io/en/latest/userguide/keywords.html

setup_requires

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (using pip if available) before processing the rest of the setup script or commands. This argument is needed if you are using distutils extensions as part of your build process; for example, extensions that process setup() arguments and turn them into EGG-INFO metadata files.

(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)

So ideally, https://github.com/redis/redis-py/pull/1780/files should have fixed it. Looks like the setup is failing due to redis being imported in setup.py before even its installed.

Processing /tmp/tmp.acCdF6XYQC/redis-py
    ERROR: Command errored out with exit status 1:
     command: /tmp/tmp.acCdF6XYQC/redis-py/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nvv2qnn6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nvv2qnn6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-nvv2qnn6/pip-egg-info
         cwd: /tmp/pip-req-build-nvv2qnn6/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-nvv2qnn6/setup.py", line 4, in <module>
        import redis
      File "/tmp/pip-req-build-nvv2qnn6/redis/__init__.py", line 1, in <module>
        from redis.client import Redis, StrictRedis
      File "/tmp/pip-req-build-nvv2qnn6/redis/client.py", line 15, in <module>
        from redis.connection import ConnectionPool, SSLConnection, UnixDomainSocketConnection
      File "/tmp/pip-req-build-nvv2qnn6/redis/connection.py", line 13, in <module>
        from packaging.version import Version
    ModuleNotFoundError: No module named 'packaging'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

redis-py/setup.py

Line 4 in 12c17bf

import redis

Creating a separate issue for this.