Installing v1.8.0 from sdist fails with old pip versions

[webknjaz]

What went wrong?

Whenever a pip version that's used doesn't support PEP 517/518, it just calls setup.py commands instead. This means that the dependencies declared in pyproject.toml don't get pre-installed.

This is what's been happening in Ansible Core CI since Friday.

How do we reproduce?

Click to expand the repro log

$ podman run --rm -it ubi8/ubi:8.6 sh -c 'set -x; dnf install -y python3-pip; python3 -V; pip3.6 install gssapi==1.8.0'
+ dnf install -y python3-pip
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Red Hat Universal Base Image 8 (RPMs) - BaseO 2.8 MB/s | 804 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppSt 6.8 MB/s | 3.0 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeR 130 kB/s |  20 kB     00:00    
Dependencies resolved.
==============================================================================
 Package             Arch   Version                     Repository       Size
==============================================================================
Installing:
 python3-pip         noarch 9.0.3-22.el8                ubi-8-appstream  20 k
Installing dependencies:
 platform-python-pip noarch 9.0.3-22.el8                ubi-8-baseos    1.6 M
 python3-setuptools  noarch 39.2.0-6.el8                ubi-8-baseos    163 k
 python36            x86_64 3.6.8-38.module+el8.5.0+12207+5c5719bc
                                                        ubi-8-appstream  19 k
Enabling module streams:
 python36                   3.6                                              

Transaction Summary
==============================================================================
Install  4 Packages

Total download size: 1.8 M
Installed size: 7.2 M
Downloading Packages:
(1/4): python3-pip-9.0.3-22.el8.noarch.rpm    230 kB/s |  20 kB     00:00    
(2/4): python36-3.6.8-38.module+el8.5.0+12207 465 kB/s |  19 kB     00:00    
(3/4): python3-setuptools-39.2.0-6.el8.noarch 1.1 MB/s | 163 kB     00:00    
(4/4): platform-python-pip-9.0.3-22.el8.noarc 4.7 MB/s | 1.6 MB     00:00    
------------------------------------------------------------------------------
Total                                         5.1 MB/s | 1.8 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                      1/1 
  Installing       : python3-setuptools-39.2.0-6.el8.noarch               1/4 
  Installing       : platform-python-pip-9.0.3-22.el8.noarch              2/4 
  Installing       : python36-3.6.8-38.module+el8.5.0+12207+5c5719bc.x8   3/4 
  Running scriptlet: python36-3.6.8-38.module+el8.5.0+12207+5c5719bc.x8   3/4 
  Installing       : python3-pip-9.0.3-22.el8.noarch                      4/4 
  Running scriptlet: python3-pip-9.0.3-22.el8.noarch                      4/4 
  Verifying        : platform-python-pip-9.0.3-22.el8.noarch              1/4 
  Verifying        : python3-setuptools-39.2.0-6.el8.noarch               2/4 
  Verifying        : python3-pip-9.0.3-22.el8.noarch                      3/4 
  Verifying        : python36-3.6.8-38.module+el8.5.0+12207+5c5719bc.x8   4/4 
Installed products updated.

Installed:
  platform-python-pip-9.0.3-22.el8.noarch                                     
  python3-pip-9.0.3-22.el8.noarch                                             
  python3-setuptools-39.2.0-6.el8.noarch                                      
  python36-3.6.8-38.module+el8.5.0+12207+5c5719bc.x86_64                      

Complete!
+ python3 -V
Python 3.6.8
+ pip3.6 install gssapi==1.8.0
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting gssapi==1.8.0
  Downloading https://files.pythonhosted.org/packages/ed/dd/562f246e551d0e5e6dc1ab0e05bafb9e5708eb8f0e99b46050dde804356e/gssapi-1.8.0.tar.gz (94kB)
    100% |████████████████████████████████| 102kB 2.7MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-o7sn4x2g/gssapi/setup.py", line 18, in <module>
        from Cython.Build import cythonize  # noqa: E402
    ModuleNotFoundError: No module named 'Cython'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-o7sn4x2g/gssapi/

Component versions (python-gssapi, Kerberos, OS / distro, etc.)

RHEL 8.6, Python 3.6, pip 9

Ideas

As a workaround, the packaging configuration could make use of the deprecated setup_requires when setuptools is detected to be below v58.3.0.

Alternatively, this package could drop the support for older Pythons that are known to bundle older pip versions.
The latest CPython 3.7 bundles pip v22.0.4: https://github.com/python/cpython/tree/3.7/Lib/ensurepip/_bundled. Though, its first stable release v3.7.0 only has pip v10.0.1 (https://github.com/python/cpython/tree/v3.7.0/Lib/ensurepip/_bundled) while PEP 517 started being implemented around pip v18.1.

We've had the same decisions to make in aiohttp around 3-4 years ago which was being complicated by Cython being an optional dependency so we've opted for shipping C-files in sdist. Although, we're now facing other problems with that due to newer Cython being needed for older releases to function under Python 3.11 that you seem to have attempted to address. FWIW, our case is easier because we can just ship wheels for GNU/Linux as well and gssapi doesn't do that.

[jborean93]

This version has been yanked from PyPI and I will push a new release under 1.8.1 that restricts python_requires >= 3.7 as that should ship with a new enough pip that supports PEP 517 out of the box. Unfortunately yanking support in pip was introduced around the same time as this stuff so they will still need to constrain the version.

Previous versions did used to include the cythonised files but it caused all sorts of problems with newer Python versions or gcc versions turning warnings into errors over time which would be fixed by newer Cython versions. I incorrect assumed that since PEP 517 support has been out since 2018 that it would be ok but seems like RHEL 8 slipped through the cracks. I'll try to request it to be deleted but there's no guarantees PyPI will accommodate that request.

I've requested the version to be deleted by the PyPI admins but I'm unsure if they will accept the request. Considering there is a python3-gssapi package from DNF on these versions and it's considered poor practice to install packages from pip onto the system Python I am hoping this isn't a common case. If using a venv they can upgrade their pip version to ensure that the yank is respected or even use Python 3.8 that is available through dnf.

[webknjaz]

I've requested the version to be deleted by the PyPI admins but I'm unsure if they will accept the request.

Closing, now that it's removed from https://pypi.org/project/gssapi/#history.

[jborean93]

Yep, the setting the delete the version was a bit lower than where I checked, 1.8.0 has been removed and 1.8.1 will include python_requires='>=3.7' which should avoid this problem.