Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix pip dependencies #169

Merged
merged 6 commits into from
Jan 19, 2024
Merged

Fix pip dependencies #169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9e4f785
remove conda-only dependencies from requirements, add matrix for cupy…
trxcllnt Jan 12, 2024
5d73396
Merge branch 'branch-0.36' of github.com:rapidsai/ucxx into fix/pip-d…
trxcllnt Jan 18, 2024
f778cc4
commit envs
trxcllnt Jan 18, 2024
1af9fb9
simplify via version star syntax
trxcllnt Jan 18, 2024
1b3d0ed
fallback to *rmm_conda
trxcllnt Jan 18, 2024
2f8b33b
Apply suggestions from code review
trxcllnt Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion conda/environments/all_cuda-118_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ dependencies:
- cuda-version=11.8
- cudatoolkit
- cudf==24.2.*
- cupy
- cupy>=12.0.0
- cxx-compiler
- cython>=3.0.0
- dask
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/all_cuda-120_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ dependencies:
- cuda-cudart-dev
- cuda-version=12.0
- cudf==24.2.*
- cupy
- cupy>=12.0.0
- cxx-compiler
- cython>=3.0.0
- dask
Expand Down
58 changes: 55 additions & 3 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ files:
- run_python
- test_cpp
- test_python
- depends_on_cupy
- depends_on_cudf
test_cpp:
output: none
includes:
Expand All @@ -27,6 +29,8 @@ files:
- cuda_version
- py_version
- test_python
- depends_on_cupy
- depends_on_cudf
checks:
output: none
includes:
Expand Down Expand Up @@ -54,6 +58,8 @@ files:
key: test
includes:
- test_python
- depends_on_cupy
- depends_on_cudf
channels:
- rapidsai
- rapidsai-nightly
Expand All @@ -76,18 +82,30 @@ dependencies:
- spdlog>=1.12.0,<1.13
build_python:
common:
- output_types: [conda]
packages:
- &rmm_conda rmm==24.2.*
- output_types: [conda, requirements, pyproject]
packages:
- *cmake_ver
- cython>=3.0.0
- ninja
- rmm==24.2.*
- output_types: conda
packages:
- scikit-build-core>=0.7.0
- output_types: [requirements, pyproject]
packages:
- scikit-build-core[pyproject]>=0.7.0
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
packages:
- rmm-cu12==24.2.*
- matrix: {cuda: "11.*"}
packages:
- rmm-cu11==24.2.*
- {matrix: null, packages: [*rmm_conda]}
checks:
common:
- output_types: [conda, requirements]
Expand Down Expand Up @@ -180,11 +198,45 @@ dependencies:
- output_types: [conda, requirements, pyproject]
packages:
- cloudpickle
- cudf==24.2.*
- cupy
- dask
- distributed
- numba>=0.57.1
- pytest
- pytest-asyncio
- pytest-rerunfailures
depends_on_cupy:
common:
- output_types: conda
packages:
- cupy>=12.0.0
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
packages:
- cupy-cuda12x>=12.0.0
- matrix: {cuda: "11.*"}
packages:
- cupy-cuda11x>=12.0.0
- {matrix: null, packages: [cupy-cuda11x>=12.0.0]}
depends_on_cudf:
common:
- output_types: conda
packages:
- &cudf_conda cudf==24.2.*
- output_types: requirements
packages:
# pip recognizes the index as a global option for the requirements.txt file
# This index is needed for rmm, cubinlinker, ptxcompiler.
- --extra-index-url=https://pypi.nvidia.com
- --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
packages:
- cudf-cu12==24.2.*
- matrix: {cuda: "11.*"}
packages:
- cudf-cu11==24.2.*
- {matrix: null, packages: [*cudf_conda]}
2 changes: 1 addition & 1 deletion python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ classifiers = [
test = [
"cloudpickle",
"cudf==24.2.*",
"cupy",
"cupy-cuda11x>=12.0.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're inconsistent in how we handle cudf/cupy fallback. This is a problem across basically all of RAPIDS, so it'd be good for us to unify our approaches.

If we need cupy-cuda11x here, we would probably want cudf-cu11 as well. For all RAPIDS packages we've been falling back to the "conda name" but we've been falling back to the "CUDA 11 pip name" for cupy. We should do one of:

  • use cudf and cupy
  • use cudf-cu11 and cupy-cuda11x

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using the "conda name" here because it works automatically when building the repos from source.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. I wondered about that. We can leave this as-is, then.

"dask",
"distributed",
"numba>=0.57.1",
Expand Down