Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip's dependency resolver doesn't resolve the conflict #9548

Closed
COfek opened this issue Feb 2, 2021 · 11 comments
Closed

pip's dependency resolver doesn't resolve the conflict #9548

COfek opened this issue Feb 2, 2021 · 11 comments
Labels
C: dependency resolution About choosing which dependencies to install S: awaiting response Waiting for a response/more information type: support User Support

Comments

@COfek
Copy link

COfek commented Feb 2, 2021

I've been trying to install a package through pip on my rpi 3 model B.
my operating system is raspbian . debian based pip version is 21.0.1 and python version is 3.7.4 the command I'm using is:

python3 -m pip install librosa
the problem is that the dependency resolver takes way too long to resolve the conflicts. and after a few hours it keeps repeating this line over and over again for hours ( I even left the installation running for 2 days over nights )

INFO: pip is looking at multiple versions of <Python from requires-Python> to determine which version is compatible with other requirements. this could take a while.
INFO: this is taking longer than usual. you might need to provide the dependency resolver with stricter constraints to reduce runtime. if you want to abort this run. you can press ctrl + c to do so.

I've tried using a stricter constraint such as adding "numpy > 1.20.0" and other stuff but now the popped up and I have no clue what I can do now.

would appreciate if anyone can provide a solution. thanks in advance.

@pradyunsg pradyunsg added the S: needs triage Issues/PRs that need to be triaged label Mar 6, 2021
@notatallshaw
Copy link
Member

Do you still have this problem or can you reproduce it?

Several improvements have happened to Pip's dependency resolver since you posted this, including just recently Pip main (28239f9) has sarugaku/resolvelib#111 and sarugaku/resolvelib#113. I haven't been able to reproduce any other long backtracking issue with these improvements.

In a clean virtual environment can you retry by installing Pip like so: python -m pip install git+https://github.com/pypa/pip.git. Or let me know if you can still reproduce at all and I will give it a go.

@pradyunsg pradyunsg added the S: awaiting response Waiting for a response/more information label Mar 28, 2023
@tillahoffmann
Copy link

tillahoffmann commented Apr 12, 2023

I've also encountered the "pip is looking at multiple versions ..." message when compiling requirements with pip-tools for the testcontainers-python package. Interestingly, this only happens on 3.11, presumably because a package is not yet available for the most recent python version or because an older version of the package required by other packages was not released for 3.11. Unfortunately, but I couldn't figure out who the culprit was.

I thought it might just be my local installation and created a GitHub workflow to compile the requirements for reproducibility (see testcontainers/testcontainers-python#331 for details). An example build is available here. The process takes less than a minute for 3.7 through 3.10. For 3.11, the process fails after about 18 mins because a package with python2 syntax is downloaded which causes a syntax error.

Unfortunately, the requirements of the testcontainers package don't constitute a minimal reproducible example, but hopefully a reproducible example is nevertheless useful.

@notatallshaw
Copy link
Member

Do you have a way to reproduce this using pip rather than pip-tools?

I tried the following:

$ python3.11 -m venv .venv
$ source .venv/bin/activate
$ pip install pip --upgrade
$ git clone https://github.com/testcontainers/testcontainers-python
$ cd testcontainers-python/
$ pip install -r requirements/3.11.txt

And it gave me the error:

ERROR: Could not find a version that satisfies the requirement codecov==2.1.12 (from versions: none)
ERROR: No matching distribution found for codecov==2.1.12

@tillahoffmann
Copy link

tillahoffmann commented Apr 12, 2023

Yes, sorry, the PR testcontainers/testcontainers-python#331 drops the codecov requirement (the python package was deleted from pip recently so the master build fails). Checking out the branch from testcontainers/testcontainers-python#331 should hopefully work. I'll also try with pip as you suggested.

@tillahoffmann
Copy link

tillahoffmann commented Apr 12, 2023

The culprit appears to be an old version of google-cloud-pubsub we depend on (our package desperately needs updating). But the following snippet reproduces the result without using pip-compile (I use the --platform flag because I'm using an M1 mac).

docker run --rm -it --platform=linux/amd64 python:3.11 pip install "google-cloud-pubsub<2"

@notatallshaw
Copy link
Member

notatallshaw commented Apr 12, 2023

The culprit appears to be an old version of google-cloud-pubsub we depend on (our package desperately needs updating). But the following snippet reproduces the result without using pip-compile (I use the --platform flag because I'm using an M1 mac).

docker run --rm -it --platform=linux/amd64 python:3.11 pip install "google-cloud-pubsub<2"

For me this creates some backtracking but on my machine it resolves in ~10 seconds, is this what you see or is it worse?

Actually this is one of the situations where the total wall time appears to get worse not better for pip main taking ~20 seconds on my machine, it's doing less backtracking but the build steps themselves are taking longer, perhaps there's been some changes to way the build isolation work,

@tillahoffmann
Copy link

tillahoffmann commented Apr 12, 2023

For me this creates some backtracking but on my machine it resolves in ~10 seconds, is this what you see or is it worse?

It takes two minutes for me but the dependencies do eventually get resolved. I've attached the full logs below.

Edit: I've updated the master branch to remove codecov in testcontainers/testcontainers-python#332. So hopefully installing from master should now work.

pip install in docker container

Collecting google-cloud-pubsub<2
  Downloading google_cloud_pubsub-1.7.2-py2.py3-none-any.whl (144 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.8/144.8 kB 1.0 MB/s eta 0:00:00
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
  Downloading google_api_core-2.11.0-py3-none-any.whl (120 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.3/120.3 kB 9.5 MB/s eta 0:00:00
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting protobuf<4.0.0dev
  Downloading protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 3.3 MB/s eta 0:00:00
Collecting googleapis-common-protos<2.0dev,>=1.56.2
  Downloading googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.6/223.6 kB 4.6 MB/s eta 0:00:00
Collecting google-auth<3.0dev,>=2.14.1
  Downloading google_auth-2.17.2-py2.py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.2/178.2 kB 12.8 MB/s eta 0:00:00
Collecting requests<3.0.0dev,>=2.18.0
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 7.3 MB/s eta 0:00:00
Collecting grpcio<2.0dev,>=1.33.2
  Downloading grpcio-1.53.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 2.3 MB/s eta 0:00:00
Collecting grpcio-status<2.0dev,>=1.33.2
  Downloading grpcio_status-1.53.0-py3-none-any.whl (5.1 kB)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 1.1 MB/s eta 0:00:00
Collecting six>=1.9.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting grpcio-status<2.0dev,>=1.33.2
  Downloading grpcio_status-1.51.3-py3-none-any.whl (5.1 kB)
  Downloading grpcio_status-1.51.1-py3-none-any.whl (5.1 kB)
  Downloading grpcio_status-1.50.0-py3-none-any.whl (14 kB)
  Downloading grpcio_status-1.49.1-py3-none-any.whl (14 kB)
INFO: pip is looking at multiple versions of grpcio to determine which version is compatible with other requirements. This could take a while.
Collecting grpcio<2.0dev,>=1.33.2
  Downloading grpcio-1.51.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 2.9 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of googleapis-common-protos[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0
  Downloading googleapis_common_protos-1.58.0-py2.py3-none-any.whl (223 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.0/223.0 kB 8.5 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of googleapis-common-protos to determine which version is compatible with other requirements. This could take a while.
  Downloading googleapis_common_protos-1.57.1-py2.py3-none-any.whl (218 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 218.0/218.0 kB 8.6 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.57.0-py2.py3-none-any.whl (217 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 218.0/218.0 kB 9.8 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.56.4-py2.py3-none-any.whl (211 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.7/211.7 kB 1.7 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.56.3-py2.py3-none-any.whl (211 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 kB 8.5 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.56.2-py2.py3-none-any.whl (211 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 kB 6.6 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.7/211.7 kB 7.2 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.5/241.5 kB 4.4 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of googleapis-common-protos[grpc] to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of googleapis-common-protos to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.
Collecting google-auth<3.0dev,>=2.14.1
  Downloading google_auth-2.17.1-py2.py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.1/178.1 kB 8.9 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of grpcio to determine which version is compatible with other requirements. This could take a while.
  Downloading google_auth-2.17.0-py2.py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.1/178.1 kB 5.0 MB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of protobuf to determine which version is compatible with other requirements. This could take a while.
Collecting protobuf<4.0.0dev
  Downloading protobuf-3.20.2-py2.py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 7.7 MB/s eta 0:00:00
  Downloading protobuf-3.19.6-py2.py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.6/162.6 kB 10.0 MB/s eta 0:00:00
  Downloading protobuf-3.19.5-py2.py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.7/162.7 kB 9.1 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of protobuf to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of grpc-google-iam-v1 to determine which version is compatible with other requirements. This could take a while.
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
  Preparing metadata (setup.py) ... done
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.5.2
  Downloading googleapis_common_protos-1.55.0-py2.py3-none-any.whl (212 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.3/212.3 kB 3.9 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.54.0-py2.py3-none-any.whl (207 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.0/208.0 kB 2.4 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.2/198.2 kB 4.7 MB/s eta 0:00:00
  Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.2/100.2 kB 3.3 MB/s eta 0:00:00
  Downloading googleapis-common-protos-1.51.0.tar.gz (35 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.50.0.tar.gz (29 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.6.0.tar.gz (35 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.10.tar.gz (37 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.9.tar.gz (37 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.8.tar.gz (33 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.7.tar.gz (33 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.6.tar.gz (33 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.5.tar.gz (33 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.4.tar.gz (33 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.3.tar.gz (28 kB)
  Preparing metadata (setup.py) ... done
  Downloading googleapis-common-protos-1.5.2.tar.gz (28 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of grpc-google-iam-v1 to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
  Downloading google_api_core-2.10.2-py3-none-any.whl (115 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.6/115.6 kB 2.3 MB/s eta 0:00:00
Collecting grpcio-status<2.0dev,>=1.33.2
  Downloading grpcio_status-1.48.2-py3-none-any.whl (14 kB)
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (197 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 197.3/197.3 kB 3.5 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 7.5 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 1.0 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 4.3 MB/s eta 0:00:00
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 4.1 MB/s eta 0:00:00
Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, idna, grpcio, charset-normalizer, certifi, cachetools, requests, googleapis-common-protos, google-auth, grpcio-status, google-api-core, grpc-google-iam-v1, google-cloud-pubsub
Successfully installed cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 google-api-core-2.10.2 google-auth-2.17.2 google-cloud-pubsub-1.7.2 googleapis-common-protos-1.59.0 grpc-google-iam-v1-0.12.6 grpcio-1.53.0 grpcio-status-1.48.2 idna-3.4 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.28.2 rsa-4.9 six-1.16.0 urllib3-1.26.15
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

@notatallshaw
Copy link
Member

It takes two minutes for me but the dependencies do eventually get resolved. I've attached the full logs below.

I diffed, I get identical output but it only takes a ~15 seconds on my machine. It is less backtracking on pip main though so hopefully it should be faster for you once pip updates to 23.1+.

Edit: I've updated the master branch to remove codecov in testcontainers/testcontainers-python#332. So hopefully installing from master should now work.

Now pip install -r requirements/3.11.txt completes very quickly for me on Python 3.11, do you see any different?

@tillahoffmann
Copy link

I diffed, I get identical output but it only takes a ~15 seconds on my machine. It is less backtracking on pip main though so hopefully it should be faster for you once pip updates to 23.1+.

Great, will make sure to stay up to date with the latest pip version.

Now pip install -r requirements/3.11.txt completes very quickly for me on Python 3.11, do you see any different?

I bit the bullet and finally updated the google-cloud-pubsub dependency that was causing the issue in testcontainers/testcontainers-python#333.

Thank you for the speedy and detailed input! 🙏

@ichard26 ichard26 added C: dependency resolution About choosing which dependencies to install type: support User Support and removed S: needs triage Issues/PRs that need to be triaged labels Apr 25, 2024
@ichard26
Copy link
Member

Unfortunately without a clear reproduction case, this issue remains unactionable. If anyone has a relatively easy to run example where pip's dependency resolver resolves suboptimally, please let us know!

@ichard26 ichard26 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 25, 2024
@ichard26
Copy link
Member

Suggestions for anyone dealing with slow resolution to 1) upgrade pip1, and 2) read https://pip.pypa.io/en/stable/topics/dependency-resolution/.

Footnotes

  1. although be wary of Pip fails to install satisfiable requirements due to bug in backjumping optimization #12317.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: dependency resolution About choosing which dependencies to install S: awaiting response Waiting for a response/more information type: support User Support
Projects
None yet
Development

No branches or pull requests

5 participants