Package hash mismatch with custom pypi under 2023.8.19 and later

[command-tab]

Issue description

Installing a package from a custom pypi registry with pipenv 2023.8.19 or later results in a hash mismatch:

[pipenv.exceptions.InstallError]: ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
[pipenv.exceptions.InstallError]:     redactedpackagename==1.2.3 from https://pypi.mycompany.com/packages/redactedpackagename-1.2.3.tar.gz#md5=d3c9c9a68fbe87a4f311ec99c61d97db (from -r /var/folders/6y/1rk0rt4j3l9gv6tjt5nxdzxm00058y/T/pipenv-x3ce9ww3-requirements/pipenv-oimob5k5-hashed-reqs.txt (line 1)):
[pipenv.exceptions.InstallError]:         Expected sha256 82118d948c6a96521a2a817a15ffb79ab88f23bc023b6295d945302013e6c1a7
[pipenv.exceptions.InstallError]:         Expected     or b18af6c93f5841f3471ef4be4f83ec0a002a8a4cf8c5742a6687844c155182f1
[pipenv.exceptions.InstallError]:              Got        93ac3f9c70d025f6ca0ee3d1d85077f28ade3d02a329d418c0c943a9a5b4afdf

I do not have reason to believe the packages have been tampered with, and the packages have not been changed, updated, or republished for well over a year.

Most telling, reverting to pipenv 2023.7.23 fixes the issue. The issue occurs under pipenv 2023.8.19, 2023.8.20, and 2023.8.21.

Expected result

I expected the custom pypi package to install without a hash mismatch.

Actual result

The above error message.

Steps to replicate

1. Publish a package on a custom/non-pypi registry

2. Create a Pipfile referencing the package:

   [[source]]
   url = "https://pypi.mycompany.com/simple"
   verify_ssl = true
   name = "mycompanypypi"
   
   [[source]]
   url = "https://pypi.python.org/simple"
   verify_ssl = true
   name = "pypi"
   
   [packages]
   arrow = "==1.2.2"
   redactedpackagename = {version="==1.2.3", index="mycompanypypi"}
   
   [requires]
   python_version = "3.11"
   

3. Run pipenv install --dev to lock dependencies and write out Pipfile.lock

4. After successfully writing out Pipfile.lock, pipenv errors as above

It's unclear to me where pipenv is getting the "Expected" hashes from when there was not a Pipfile.lock on disk until pipenv created it, after which it immediately disagrees with itself about what the hash ought to be. It seems as if the logic that writes Pipfile.lock is doing something incorrectly, and the system that verifies the actual hash is working correctly; For comparison's sake, if I fetch the redactedpackagename archive from mycompanypypi and sha256 it myself, I get the 93ac... hash that pipenv "Got".

Of note:

• If I remove the redactedpackagename line from Pipfile and re-run the steps (thus only installing arrow from pypi), everything succeeds.
• If I re-add the redactedpackagename line and re-run, it fails again as expected.
• If I downgrade pipenv to 2023.7.23 and re-run, everything succeeds. So, it appears there was some change after pipenv 2023.7.2 that resulted in this breakage.

---

(I've modified this support info a little to redact some company-specific things, but I don't think I've done it in a way that compromises its purpose.)

$ pipenv --support

Pipenv version: '2023.8.21'

Pipenv location: '/Users/callen/.pyenv/versions/3.11.3/lib/python3.11/site-packages/pipenv'

Python location: '/Users/callen/.pyenv/versions/3.11.3/bin/python3.11'

OS Name: 'posix'

User pip version: '23.2.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.3',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:53 PDT '
                     '2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6020',
 'python_full_version': '3.11.3',
 'python_version': '3.11',
 'sys_platform': 'darwin'}

System environment variables:

• SHELL
• LSCOLORS
• PYENV_HOOK_PATH
• PIPENV_VENV_IN_PROJECT
• PYENV_SHELL
• LESS
• XPC_FLAGS
• HISTCONTROL
• TERM_PROGRAM_VERSION
• __CFBundleIdentifier
• SSH_AUTH_SOCK
• XDG_CONFIG_HOME
• TERM_SESSION_ID
• PYENV_VIRTUALENV_DISABLE_PROMPT
• EDITOR
• PYENV_VERSION
• PWD
• LOGNAME
• MANPATH
• HOME
• LANG
• TMPDIR
• SSH_CLIENT_USER
• PIPENV_IGNORE_VIRTUALENVS
• PYENV_DIR
• PIPENV_HIDE_EMOJIS
• TERM
• USER
• MANPAGER
• SHLVL
• GIT_EDITOR
• PAGER
• XPC_SERVICE_NAME
• PYENV_ROOT
• PATH
• PIPENV_PYTHON
• __CF_USER_TEXT_ENCODING
• TERM_PROGRAM
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1
• PIPENV_IGNORE_VIRTUALENVS: 1
• PIPENV_HIDE_EMOJIS: 1
• PIPENV_PYTHON: /Users/callen/.pyenv/shims/python

Debug–specific environment variables:

• PATH: /Users/callen/.pyenv/versions/3.10.11/bin:/Users/callen/.pyenv/versions/3.11.3/bin:/Users/callen/.pyenv/versions/3.8.10/bin:/Users/callen/.pyenv/versions/3.11.3/bin:/opt/homebrew/Cellar/pyenv/2.3.22/libexec:/opt/homebrew/Cellar/pyenv/2.3.22/plugins/python-build/bin:/Users/callen/.cargo/bin:/Users/callen/.pyenv/shims:/usr/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/callen/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
• SHELL: /bin/zsh
• EDITOR: /usr/local/bin/nova -w
• LANG: en_US.UTF-8
• PWD: /Users/callen/Desktop/foo

---

Contents of Pipfile ('/Users/callen/Desktop/foo/Pipfile'):

[[source]]
url = "https://pypi.mycompany.com/simple"
verify_ssl = true
name = "mycompanypypi"

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
arrow = "==1.2.2"
redactedpackagename = {version="==1.2.3", index="mycompanypypi"}

[requires]
python_version = "3.11"

Contents of Pipfile.lock ('/Users/callen/Desktop/foo/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "60b108214c501bf12ef91090ead2b13524adbb60333222b450f88390a093909d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "mycompanypypi",
                "url": "https://pypi.mycompany.com/simple",
                "verify_ssl": true
            },
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "arrow": {
            "hashes": [
                "sha256:05caf1fd3d9a11a1135b2b6f09887421153b94558e5ef4d090b567b47173ac2b",
                "sha256:d622c46ca681b5b3e3574fcb60a04e5cc81b9625112d5fb2b44220c36c892177"
            ],
            "index": "mycompanypypi",
            "markers": "python_version >= '3.6'",
            "version": "==1.2.2"
        },
        "python-dateutil": {
            "hashes": [
                "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86",
                "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'",
            "version": "==2.8.2"
        },
        "redactedpackagename": {
            "hashes": [
                "sha256:82118d948c6a96521a2a817a15ffb79ab88f23bc023b6295d945302013e6c1a7",
                "sha256:b18af6c93f5841f3471ef4be4f83ec0a002a8a4cf8c5742a6687844c155182f1"
            ],
            "index": "mycompanypypi",
            "version": "==1.3.3"
        },
        "six": {
            "hashes": [
                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'",
            "version": "==1.16.0"
        }
    },
    "develop": {}
}

Thank you for taking the time to look at this issue, and for continuing to work on pipenv ❤️

[command-tab]

This seems noteworthy:

Digging through the pipenv 2023.8.21 source and adding some logging to understand how it works, I noticed that get_file_hash in project.py was downloading archives from https://pypi.mycompany.com/simple/packages/... to produce hashes of the wheel and tgz archives. The correct URL for archives is just https://pypi.mycompany.com/packages/..., though the index does have a simple index page at /simple.

What was actually being hashed by get_file_hash was the contents of the unique 404 web page for each archive. Those hashes of HTML content were then being compared against the hash of the correct wheel archive bytes, resulting in the mismatch.

Removing /simple from the Pipfile makes everything install correctly under pipenv 2023.8.21:

[[source]]
url = "https://pypi.mycompany.com"
verify_ssl = true
name = "mycompanypypi"

Did something change between 2023.7.23 and 2023.8.19 regarding downloading of packages? Maybe the previous implementation was omitting /simple from custom pypi index URLs and the new one doesn't?

[cecedille1]

I've had this issue, my pypi has tokens in its URL, the index url is https://pypi.company.com/pypi/abcdef1234 and the package URLs are like https://pypi.company.com/pypi/abcdef1234/package. This page generates content with absolute URL in links (https://bucket.amazon.com/package/package-version.whl).

With the previous version 2023.7.23 it follows the URL with the usual resolve rules. But with 2023.8.19 and later it appends the path to the current page, meaning the actual download is https://pypi.company.com/pypi/abcdef1234https://bucket.amazon.com/package/package-version.whl.

I tried to generate an URL relative to the root /redirect/abcdef1234/package/package-version.whl that does a redirect but alas it also appends and the dowload URL is https://pypi.company.com/pypi/abcdef1234/redirect/abcdef1234/package/package-version.whl

[matteius]

What was actually being hashed by get_file_hash was the contents of the unique 404 web page for each archive.

Oh wow ... 🤦‍♂️

Some backstory here -- I originally added the code path to start checking private pypis to speed up locking -- and this will be a big deal with something like pytorch once the work backing pytorch/builder#1433 gets completed because instead of downloading 8x 3.5 GB wheels to get the hashes, it will just get them from the index URLs.

That being said, clearly I introduced some edge cases that I didn't foresee. Totally open to PRs to help clean those up, I can guarantee I won't be able to look at this immediately.

With the previous version 2023.7.23 it follows the URL with the usual resolve rules. But with 2023.8.19 and later it appends the path to the current page, meaning the actual download is

This also surprises me, but seems possible -- from my perspective of what I was seeing in my testing that lead to me changing that is we were calling to get the hash with just the trailing /simple/uri.whatever without the root domain of the test pypi server so it wasn't gathering the hashes correctly in my test example. 🤦 🤦

Sounds like there are a couple things to sort out here:

• the 404s being treated as match
• the double URL edge case
• something else?

[matteius]

I think I got to the bottom of this issue (I hope for all cases), could you check out if #5866 resolves the issue for your case?

[command-tab]

Thank you! I’ll give the patch a run through today and report back.

[command-tab]

Success! The change works great, and so does release 2023.8.25. Thank you for resolving this!