pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000 error

[sreelathasankaranarayanan]

Description

We are consistently see this error in our pip install jobs, can you please suggest if there is a way to reduce the backtracking of dependencies to prevent this error from happenning. We started noticing this problem consistently since Jun 6. Has there been any changes with pip 24.0 or setuptools 70.0.0 or a combination that needs to be considered?

We notice this problem in python version <3.11 and in python version 3.12 onwards we do not see this error.

ERROR: Exception:
Traceback (most recent call last):
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Expected behavior

pip installation should succeed without resolution too deep errors

pip version

24.0

Python version

3.9

OS

Ubuntu 20.0

How to Reproduce

pip install <package_name> --upgrade

Output

ERROR: Exception:
Traceback (most recent call last):
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Can you give a reproducible example? Very likely there's been a change in the dependency tree that's hitting a pathological use case. I might be able to provide an additional constraint on your requirements as a workaround to this error.

If you would like, I have a branch where I have tried to significantly reduce the chance of this happening, here is an example how how to install #12305 (comment). Feedback on if this branch helped or not would be helpful to me.

[sreelathasankaranarayanan]

Thanks @notatallshaw , let me give that a try, so you are suggesting to install pip from your branch. Is there any restriction on the version of setuptools to be used?

[sreelathasankaranarayanan]

@notatallshaw that worked like a charm! But how do we consume this, is it going to be merged to pip:main and available as a version of pip that we can directly install?

[notatallshaw]

My hope is eventually it will be merged into pip main, but I have not had chance to work on it recently, and getting it into pip requires multiple steps.

If you can give me your requirements that cause this issue for you, I can likely identify how to fix them so it works with regular pip.

[sreelathasankaranarayanan]

@notatallshaw our project has the following dependencies in setup.cfg for our project is internal to our org.

auditree-arboretum>=0.5.0
deprecated>=1.2.10
ibmcloud_tools[cos]>=2.36.0
python-dateutil<3.0.0
SoftLayer>=5.7.2
sdcclient>=0.10.0
pygithub
kubernetes>=21.7.0
pytenable
strenum
ibm-platform-services
ibm_cloud_sdk_core

Since yesterday we started to see our build failing with pip trying to resolve dependency by getting a faulty old version of pytenable (1.4.2), this halted our installation.

Then we tried to pin the version of pytenable to >=1.4.13, but this helped get rid of the failure with pytenable 1.4.2, but then we started to consistently see the pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000 errors and looking at the logs it is not always throwing this error after getting a certain package or while trying to check dependencies for a specific package.

We are using pip version 24.0 and setuptools 70.0.0, we tried to downgrade the versions of these tools as well, but to no avail. Really appreciate your help with this.

This is the dependency tree got by running pipdeptree from. my project
deptree.txt

[pfmoore]

I just tried this, but there's no project called ibmcloud_tools on PyPI. Could you provide a full, reproducible example?

[notatallshaw]

I was able to reproduce by looking at the deptree and substituting in the non-public packages with their dependencies, I was testing on Python 3.11 Linux and I found I had to add a few extra lower bounds so pip didn't try to build a package too old that building just failed:

auditree-arboretum>=0.5.0
deprecated>=1.2.10
configparser
python-dateutil
PyYAML
requests
ilcli>=0.3.2
PyJWT>=2.6.0
python-dateutil
requests>=2.28.2
configparser
python-dateutil
PyYAML
requests
python-dateutil<3.0.0
SoftLayer>=5.7.2
sdcclient>=0.10.0
pygithub>=1.58.0
kubernetes>=21.7.0
pytenable>=1.4.12
strenum>=0.4.10 
ibm-platform-services>=0.30.3 
ibm_cloud_sdk_core>=3.16.1
GitPython
ibm-cos-sdk >=2.12.1
ibm-platform-services >=0.22.5

There's probably a simpler set of requirements to reproduce this, but it takes a long time to test reproducing.

First, let's check if this resolves quickly from two days ago, testing using pypi-timemachine I set the date to "2024-06-05", and indeed it resolved very quickly. So the only thing that changed was the ecosystem, not pip or setuptools.

Looking at what packages downloaded instead of using cached versions when I ran the version against "2024-06-05" the big thing that stood out to me was sdcclient which had a release on the 6th June.

If I put an upper bound on sdcclient sdcclient>=0.10.0,<0.17.2 then indeed it quickly resolves again! So this is our culprit but it is not satisfying to put an upper bound on it and leave it at that.

Comparing the metadata from the two releases:

• https://files.pythonhosted.org/packages/68/45/837a47a66ee48f38996aafc5c12f7c00c5aa9f62223e3e9e7aecbf0ae71a/sdcclient-0.17.2-py3-none-any.whl.metadata
• https://files.pythonhosted.org/packages/31/c8/645fae65dba11dbeb7d49f4b134bbaa7ac6f2bf294a8e27d26e23c19af7d/sdcclient-0.17.1-py3-none-any.whl.metadata

There are two significant changes requests-toolbelt (>=0.9.1,<0.10.0) to requests-toolbelt (>=1.0.0,<2.0.0) and urllib3 (>=1.26.0,<2.0.0) to urllib3 (>=2.2.1,<3.0.0), we can find the likely culprit by swapping out sdcclient>=0.10.0 with either of these two. And indeed, the culprit reveals itself to be urllib3 (>=2.2.1,<3.0.0), swapping that in produces the error again.

A little further investigation revealed that ibm-cos-sdk-core always depends on some urllib3 <1.27 and therefore is not comatible with sdcclient 0.17.2.

So your solution for now is to:

1. Put an upper bound sdcclient<0.17.2
2. And, reach out to ibm-cos-sdk-core and ask them to remove urllib3 <1.27 unless they know for sure newer versions break their framework, and in general remind library authors not to eagerly put upper bounds on dependencies unless they have good reason (https://iscinumpy.dev/post/bound-version-constraints/) as it generally causes long term damage to the ecosystem
3. Politely ask sdcclient to loosen their requirements also, urllib3 1.x is still supported, are they actually already depending on 2.x features?

Hope that helps.

Pip is particularly bad at handeling these situations where it encounters a bad upper bound deep into the dependency tree. But this is a duplicate of #12305 and hopefully we can eventually get pip much better.

[sreelathasankaranarayanan]

@notatallshaw Thanks a ton for helping debug the problem for us. It immensely helped us to understand the whole problem we faced wrt the sudden change in our project's behaviour in dependency resolution without any changes from our end.
Agree that this is a duplicate of #12305 and I hope your fix for the same can make it to a future pip version soon.

[ichard26]

Merging into #12305.

[notatallshaw]

Discussed with @ichard26, I'm going to repopen this issue because I happen to know that #12305 is about to be fixed, but unfortunately this particular ResolutionTooDeep is not 😞

[notatallshaw]

Further, I'm directly tracking this resolution as changes / improvements are made to the resolver. See: https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks/blob/main/scenarios/problematic.toml#L17

[jiffyclub]

I am getting this with pip install apache-airflow==2.7.3 google-cloud-logging==3.10.0. pip version is 24.3.1, Python version is 3.11.10.

Adding opentelemetry-proto==1.27.0 to the install makes it work, it looks like something in opentelemetry-proto 1.28.0 is causing the problem.

[notatallshaw]

Thanks for the example, I will add it too the list of known problematic scenarios, and check if any of the optimizations I'm currently working on will fix it.

[notatallshaw]

I am getting this with pip install apache-airflow==2.7.3 google-cloud-logging==3.10.0. pip version is 24.3.1, Python version is 3.11.10

This appears to be caused by a bug in resolvelib's backjumping optimization, which will be fixed when #13001 lands (I am working on getting it out of draft now).