pip does not skip platform-dependent ignores when installing from cache

[jontwo]

Environment

• pip version: 20.0.2
• Python version: 2.7.15 (yes, I know)
• OS: Scientific Linux 3.10.0-957.27.2.el7.x86_64

Description

Platform dependencies are ignored when installing from cache.

This may not be a pip issue, if so, apologies. One of our dependencies (watchdog) has Mac-only dependencies. Our CI builds started to fail because pip was no longer ignoring them.

Log from a passing build:

Collecting watchdog
  Downloading watchdog-0.10.0.tar.gz (94 kB)
  Ignoring pyobjc-framework-Cocoa: markers 'sys_platform == "darwin"' don't match your environment
  Ignoring pyobjc-framework-FSEvents: markers 'sys_platform == "darwin"' don't match your environment

Log from a failing build:

Processing /home/jba/.cache/pip/wheels/fe/9f/95/60273d891e46fc8351c54b15cd5817ddf82e3420012eba69b3/watchdog-0.10.0-py2-none-any.whl

This build then failed while trying to install pyobjc-framework-FSEvents.

Expected behavior

pip should continue to ignore Mac-only dependencies.

How to Reproduce

1. pip install watchdog
2. pip install watchdog again.
3. An error occurs: IOError: [Errno 2] No such file or directory: '/System/Library/CoreServices/SystemVersion.plist'

The second attempt installs from cache and does not skip ignored packages.

Output

$ pip install watchdog
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting watchdog
  Downloading https://files.pythonhosted.org/packages/01/01/cc9b2fd111e19687dd42155f066718bccb414c8a728eb7df0b229742caf7/watchdog-0.10.0.tar.gz (94kB)
     |████████████████████████████████| 102kB 2.9MB/s 
  Ignoring pyobjc-framework-Cocoa: markers 'sys_platform == "darwin"' don't match your environment
  Ignoring pyobjc-framework-FSEvents: markers 'sys_platform == "darwin"' don't match your environment
Collecting pathtools>=0.1.1
  Using cached https://files.pythonhosted.org/packages/e7/7f/470d6fcdf23f9f3518f6b0b76be9df16dcc8630ad409947f8be2eb0ed13a/pathtools-0.1.2.tar.gz
Building wheels for collected packages: watchdog, pathtools
  Building wheel for watchdog (setup.py) ... done
  Created wheel for watchdog: filename=watchdog-0.10.0-cp27-none-any.whl size=73911 sha256=adc84a0fada2a345c9c583ae5d7c46d3b5059f56a097b0049aa69be7ef7c6a25
  Stored in directory: /home/jba/.cache/pip/wheels/d4/74/90/4e0d4d5050776ce9a03e8a2c873e523b0149bf6412546dfc3f
  Building wheel for pathtools (setup.py) ... done
  Created wheel for pathtools: filename=pathtools-0.1.2-cp27-none-any.whl size=8606 sha256=d3c9c7f3cab423dbb16a3f1067efa0573528fb9abafe36ff8068bf30258f8407
  Stored in directory: /home/jba/.cache/pip/wheels/0b/04/79/c3b0c3a0266a3cb4376da31e5bfe8bba0c489246968a68e843
Successfully built watchdog pathtools
ERROR: watchdog 0.10.0 requires pyobjc-framework-Cocoa>=4.2.2, which is not installed.
ERROR: watchdog 0.10.0 requires pyobjc-framework-FSEvents>=4.2.2, which is not installed.
Installing collected packages: pathtools, watchdog
Successfully installed pathtools-0.1.2 watchdog-0.10.0
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(farm27) jba @ ~/work/farmlib (T6108)
└─ $ pip install watchdog
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: watchdog in /home/jba/venv/farm27/lib/python2.7/site-packages (0.10.0)
Requirement already satisfied: pathtools>=0.1.1 in /home/jba/venv/farm27/lib/python2.7/site-packages (from watchdog) (0.1.2)
Collecting pyobjc-framework-FSEvents>=4.2.2
  Downloading https://files.pythonhosted.org/packages/a8/32/51f1dd0aee282142c5fbca9a48d3a06c5b464e2f6e523e7f23813c4c24fc/pyobjc-framework-FSEvents-5.3.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /home/jba/venv/farm27/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vVqzQq/pyobjc-framework-FSEvents/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vVqzQq/pyobjc-framework-FSEvents/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-vVqzQq/pyobjc-framework-FSEvents/pip-egg-info
         cwd: /tmp/pip-install-vVqzQq/pyobjc-framework-FSEvents/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vVqzQq/pyobjc-framework-FSEvents/setup.py", line 22, in <module>
        [ "Modules/_callbacks.m" ],
      File "pyobjc_setup.py", line 392, in Extension
        os_level = get_os_level()
      File "pyobjc_setup.py", line 206, in get_os_level
        pl = plistlib.readPlist('/System/Library/CoreServices/SystemVersion.plist')
      File "/usr/lib64/python2.7/plistlib.py", line 75, in readPlist
        pathOrFile = open(pathOrFile)
    IOError: [Errno 2] No such file or directory: '/System/Library/CoreServices/SystemVersion.plist'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[jontwo]

Happy to file this report downstream (i.e. with watchdog or pyobjc-framework) if this is not something pip should be handling.

[uranusjr]

Would it be possible to obtain a copy of watchdog-0.10.0-cp27-none-any.whl in the cache? That would help locate the problem.

---

Edit: Also, is the output you posted for a successful or unsuccessful build? It does not use the cache or show the IOError, so I assume is the successful one? Does an unsuccessful build emit any logging like this?

[jontwo]

@uranusjr No problem, here is the wheel from the cache on the CI server (zipped as Github doesn't like .whl files).
watchdog-0.10.0-py2-none-any.whl.zip
The output is from my local machine, but the stack trace on the CI server was the same. I can post that too if you need.

[uranusjr]

Thanks! Reading the wheel metadata, I know why the platform-dependent requirements are installed:

Requires-Dist: pyobjc-framework-Cocoa (>=4.2.2)
Requires-Dist: pyobjc-framework-FSEvents (>=4.2.2)

Those are not properly marked as platform-dependent. The problem then becomes why your wheel is built like that. Reading watchdog’s source, I see it is using proper markers, and so should produce a correct wheel. Maybe the problem is in the build chain? Maybe you could try removing the cached wheel, pip install -U setuptools wheel to upgrade the build chain, and see if the newly-build wheel would work.

[jontwo]

Thanks. We've got around the problem by changing our requirements to watchdog < 0.10.0 as we don't need a specific version. Should I raise a bug with watchdog? It may be due to a change made at 0.10.0.

[uranusjr]

Yeah, I guess they might have some ideas about what happened in that particular version.

[sbidoul]

It looks like there is nothing to do in pip wrt this issue, so I'll let the bot close this unless there is additional information coming up.

[jontwo]

This can be closed now, as the issue has been resolved by watchdog. Thanks for your help!

[BoboTiG]

Just to complete the information, we fixed watchdog by moving dependencies from setup.py to setup.cfg. Then the issue was gone.

Also I did not manage to reproduce the issue on my machine (GNU/Linux). So perhaps is this a specific issue on a specific machine.

[reversefold]

Doing some more debugging it seems the problem is with an old version of either setuptools or wheel. My builds are currently pegging pex==1.3.2 which depends on older versions of setuptools and wheel.
pex-1.3.2 setuptools-33.1.1 wheel-0.29.0
Upgrading pex (and hence setuptools and wheel) seems to fix the problem.

[BoboTiG]

Did you manage to know the exact setuptools version that fixed the issue? If so, I could add it as a minimum requirement when installing the module.