pip install -r requirements-dev.txt hangs during resolving of deps between packages

[prutskov]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.2 LTS
• Modin installed from (source or binary): source
• Modin version: 7a81588
• Python version: 3.8.12
• Exact command to reproduce:

pip install -r requirements-dev.txt

Describe the problem

Installing of requirements for Modin development via pip hangs during resolving dependencies in clean environment. But env can be successfully created (but it will take a lot of time) with conda approach.

Source code / logs

Collecting bokeh!=2.0.0,>=1.0.0
  Using cached bokeh-2.4.0-py3-none-any.whl (18.4 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached bokeh-2.3.3.tar.gz (10.7 MB)
  Using cached bokeh-2.3.2.tar.gz (10.7 MB)
  Using cached bokeh-2.3.1.tar.gz (10.6 MB)
  Using cached bokeh-2.3.0.tar.gz (10.6 MB)
  Using cached bokeh-2.2.3.tar.gz (8.8 MB)
  Using cached bokeh-2.2.2.tar.gz (8.8 MB)
INFO: pip is looking at multiple versions of botocore to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of bokeh to determine which version is compatible with other requirements. This could take a while.
  Using cached bokeh-2.2.1.tar.gz (8.8 MB)
  Using cached bokeh-2.2.0.tar.gz (8.8 MB)
  Using cached bokeh-2.1.1.tar.gz (19.3 MB)
  Using cached bokeh-2.1.0.tar.gz (19.3 MB)
  Using cached bokeh-2.0.2.tar.gz (8.6 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached bokeh-2.0.1.tar.gz (8.6 MB)
  Using cached bokeh-1.4.0.tar.gz (32.4 MB)
  Using cached bokeh-1.3.4.tar.gz (17.8 MB)
  Using cached bokeh-1.3.2.tar.gz (17.8 MB)
  Using cached bokeh-1.3.1.tar.gz (17.9 MB)
  Using cached bokeh-1.3.0.tar.gz (17.8 MB)
  Using cached bokeh-1.2.0.tar.gz (17.6 MB)
  Using cached bokeh-1.1.0.tar.gz (17.5 MB)
  Using cached bokeh-1.0.4.tar.gz (16.3 MB)
  Using cached bokeh-1.0.3.tar.gz (16.3 MB)
  Using cached bokeh-1.0.2.tar.gz (16.2 MB)
  Using cached bokeh-1.0.1.tar.gz (16.1 MB)
  Using cached bokeh-1.0.0.tar.gz (16.1 MB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
  Using cached attrs-20.2.0-py2.py3-none-any.whl (48 kB)

[YarShev]

We use --use-deprecated=legacy-resolver as a solution in our CI.

modin/.github/workflows/push-to-master.yml

Line 118 in 7a81588

- run: pip install -r requirements-dev.txt --use-deprecated=legacy-resolver

Related issue #3474.

[prutskov]

In this case we should add note about this in our contributing guidelines or fix that by some another way.

[prutskov]

When we use this way

modin/.github/workflows/push-to-master.yml

Line 118 in 7a81588

- run: pip install -r requirements-dev.txt --use-deprecated=legacy-resolver

we get the next error message during env installing:

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
pytest-cov 3.0.0 requires coverage[toml]>=5.2.1, but you'll have coverage 4.5.4 which is incompatible.

[YarShev]

@anmyachev , did you encounter the issue when you added the changes?

[anmyachev]

@anmyachev , did you encounter the issue when you added the changes?

No.

[YarShev]

@prutskov , it probably depends on pip version. Will you look into this?

[prutskov]

@YarShev Yes, I will look

[Garra1980]

I even can't start the process
ERROR: Could not find a version that satisfies the requirement pandas==1.4.1 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5)
ERROR: No matching distribution found for pandas==1.4.1

What am I doing wrong?

[prutskov]

I even can't start the process ERROR: Could not find a version that satisfies the requirement pandas==1.4.1 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5) ERROR: No matching distribution found for pandas==1.4.1

What am I doing wrong?

Looks like you use python3.7 in your env. The latest pandas version which supports py3.7 is 1.3.5.

[Garra1980]

Thanks! Now I can reproduce it

[Garra1980]

We have coverage<5.0 but later I see that we need
Collecting coverage[toml]>=5.2.1

Eg relaxing coverage requirement and removing matplotlib (otherwise installation fails) leads to successfull installation

[YarShev]

@modin-project/modin-core, what is the reason for coverage<5.0? @Garra1980, what the error do you occur with matplotlib?

[Garra1980]

I tried

diff --git a/requirements-dev.txt b/requirements-dev.txt
index 55ed2d4a..c5bb56ed 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -14,14 +14,14 @@ scipy
 s3fs>=2021.8
 pytest
 pytest-benchmark
-coverage<5.0
+coverage
 pytest-cov
 pytest-xdist
 feather-format
 lxml
 openpyxl
 xlrd
-matplotlib<=3.2.2
+matplotlib
 sqlalchemy>=1.4.0
 msgpack
 pandas_gbq

and installation was successful

[YarShev]

@Garra1980, could you open a PR with these changes to see how tests go?

[prutskov]

The PRs which added pins:

1. coverage Change Coverage version to avoid sqlite3 errors #916
2. matplotlib Restrict matlpotlib version while bug #1747 is investigated #1748

I confirm, that unpinning helps with successful environment dependencies creation.