Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

split up CUDA-suffixed dependencies in dependencies.yaml #16183

Merged
merged 24 commits into from
Jul 24, 2024
Merged

split up CUDA-suffixed dependencies in dependencies.yaml #16183

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f409298
split up CUDA-suffixed dependencies in dependencies.yaml
jameslamb Jul 2, 2024
ecb59b2
typo
jameslamb Jul 2, 2024
3bdae9f
Merge branch 'branch-24.08' into suffix-split
jameslamb Jul 3, 2024
694a07f
pynvjitlink is not built from source in devcontainers
jameslamb Jul 3, 2024
4982781
all the projects depending on something suffixed need this matrix entry
jameslamb Jul 3, 2024
3c176e9
merge branch-24.08
jameslamb Jul 22, 2024
dd5c3e3
try overridding matrix entry via containerEnv
jameslamb Jul 22, 2024
6342529
Merge branch 'branch-24.08' into suffix-split
jameslamb Jul 22, 2024
de682df
fix templating
jameslamb Jul 22, 2024
868d735
Merge branch 'suffix-split' of github.com:jameslamb/cudf into suffix-…
jameslamb Jul 22, 2024
e98f8a1
revert some changes
jameslamb Jul 22, 2024
08059f1
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into suffix-s…
jameslamb Jul 22, 2024
d556aaf
Merge branch 'branch-24.08' into suffix-split
jameslamb Jul 23, 2024
255f529
no need to override config settings, libraries are installed with --n…
jameslamb Jul 23, 2024
3578294
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into suffix-s…
jameslamb Jul 23, 2024
743003e
Merge branch 'suffix-split' of github.com:jameslamb/cudf into suffix-…
jameslamb Jul 23, 2024
627d9a3
fix comment on pynvjitlink
jameslamb Jul 24, 2024
9e58c66
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into suffix-s…
jameslamb Jul 24, 2024
193bc88
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into suffix-s…
jameslamb Jul 24, 2024
035213e
remove pynvjitlink special case
jameslamb Jul 24, 2024
f0f90d0
consolidate cuda_suffixed=false conditions, stop naming anchors _conda
jameslamb Jul 24, 2024
034aac8
one more anchor
jameslamb Jul 24, 2024
2061a24
catch more cases in update-version
jameslamb Jul 24, 2024
1416fa7
more update-version changes
jameslamb Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ci/release/update-version.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,18 @@ done
# README.md update
sed_runner "s/version == ${CURRENT_SHORT_TAG}/version == ${NEXT_SHORT_TAG}/g" README.md
sed_runner "s/cudf=${CURRENT_SHORT_TAG}/cudf=${NEXT_SHORT_TAG}/g" README.md
sed_runner "s/cudf=${CURRENT_SHORT_TAG}/cudf=${NEXT_SHORT_TAG}/g" python/cudf_polars/docs/overview.md
sed_runner "s/branch-${CURRENT_SHORT_TAG}/branch-${NEXT_SHORT_TAG}/g" python/cudf_polars/docs/overview.md

# Libcudf examples update
sed_runner "s/CUDF_TAG branch-${CURRENT_SHORT_TAG}/CUDF_TAG branch-${NEXT_SHORT_TAG}/" cpp/examples/versions.cmake

# CI files
for FILE in .github/workflows/*.yaml; do
for FILE in .github/workflows/*.yaml .github/workflows/*.yml; do
sed_runner "/shared-workflows/ s/@.*/@branch-${NEXT_SHORT_TAG}/g" "${FILE}"
sed_runner "s/dask-cuda.git@branch-[^\"\s]\+/dask-cuda.git@branch-${NEXT_SHORT_TAG}/g" ${FILE};
done
sed_runner "s/branch-[0-9]+\.[0-9]+/branch-${NEXT_SHORT_TAG}/g" ci/test_wheel_cudf_polars.sh

# Java files
NEXT_FULL_JAVA_TAG="${NEXT_SHORT_TAG}.${PATCH_PEP440}-SNAPSHOT"
Expand Down
71 changes: 50 additions & 21 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ dependencies:
common:
- output_types: conda
packages:
- &rmm_conda rmm==24.8.*,>=0.0.0a0
- &rmm_unsuffixed rmm==24.8.*,>=0.0.0a0
- pip
- pip:
- git+https://github.com/python-streamz/streamz.git@master
Expand All @@ -343,13 +343,17 @@ dependencies:
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
packages: &build_python_packages_cu12
- matrix:
Copy link
Contributor

@bdice bdice Jul 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simplify this using DFG's "match first" approach? This is equivalent, and far fewer lines. If cuda_suffixed is false, we can use a fallback matrix.

          - matrix:
              cuda: "12.*"
              cuda_suffixed: "true"
            packages:
              - rmm-cu12==24.8.*,>=0.0.0a0
          - matrix:
              cuda: "11.*"
              cuda_suffixed: "true"
            packages:
              - rmm-cu11==24.8.*,>=0.0.0a0
          - matrix:
            packages:
              - *rmm_conda

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can also rename rmm_conda to rmm_unsuffixed? It's not just for conda anymore.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah totally support both of these suggestions. I'll do that here and across the other PRs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I just pushed a commit that does both of these.

  • got rid of almost all the cuda_suffixed: "false" blocks (one is still needed for the pynvjitlink vs. cubinlink/ptxcompiler split)
  • renamed all the *_conda anchors to *_unsuffixed

I tested like this, and saw the things I expected:

rapids-dependency-file-generator \
    --file-key all \
    --output requirements \
    --matrix 'cuda=11.8;cuda_suffixed=true'

rapids-dependency-file-generator \
    --file-key all \
    --output requirements \
    --matrix 'cuda=11.8;cuda_suffixed=false'

rapids-dependency-file-generator \
    --file-key all \
    --output requirements \
    --matrix 'cuda=12.2;cuda_suffixed=true'

rapids-dependency-file-generator \
    --file-key all \
    --output requirements \
    --matrix 'cuda=12.2;cuda_suffixed=false'

cuda: "12.*"
cuda_suffixed: "true"
packages:
- rmm-cu12==24.8.*,>=0.0.0a0
- matrix: {cuda: "11.*"}
packages: &build_python_packages_cu11
- matrix:
cuda: "11.*"
cuda_suffixed: "true"
packages:
- rmm-cu11==24.8.*,>=0.0.0a0
- {matrix: null, packages: [*rmm_conda] }
- {matrix: null, packages: [*rmm_unsuffixed]}
libarrow_build:
common:
- output_types: conda
Expand Down Expand Up @@ -567,7 +571,7 @@ dependencies:
- typing_extensions>=4.0.0
- output_types: conda
packages:
- *rmm_conda
- *rmm_unsuffixed
- output_types: requirements
packages:
# pip recognizes the index as a global option for the requirements.txt file
Expand All @@ -588,23 +592,40 @@ dependencies:
matrices:
- matrix: {cuda: "12.*"}
packages:
- pynvjitlink>=0.0.0a0
- &pynvjitlink_unsuffixed pynvjitlink>=0.0.0a0
- matrix: {cuda: "11.*"}
packages:
- cubinlinker
- ptxcompiler
- &cubinlinker_unsuffixed cubinlinker
- &ptxcompiler_unsuffixed ptxcompiler
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
- matrix:
cuda: "12.*"
cuda_suffixed: "true"
packages:
- rmm-cu12==24.8.*,>=0.0.0a0
- pynvjitlink-cu12>=0.0.0a0
- matrix: {cuda: "11.*"}
- matrix:
cuda: "12.*"
cuda_suffixed: "false"
packages:
- *rmm_unsuffixed
- *pynvjitlink_unsuffixed
- matrix:
cuda: "11.*"
cuda_suffixed: "true"
packages:
- rmm-cu11==24.8.*,>=0.0.0a0
- cubinlinker-cu11
- ptxcompiler-cu11
- {matrix: null, packages: [cubinlinker, ptxcompiler, *rmm_conda]}
- matrix:
cuda: "11.*"
cuda_suffixed: "false"
packages: &run_cudf_cu11_unsuffixed
- *cubinlinker_unsuffixed
- *ptxcompiler_unsuffixed
- *rmm_unsuffixed
- {matrix: null, packages: *run_cudf_cu11_unsuffixed}
run_cudf_polars:
common:
- output_types: [conda, requirements, pyproject]
Expand Down Expand Up @@ -706,7 +727,7 @@ dependencies:
common:
- output_types: conda
packages:
- &cudf_conda cudf==24.8.*,>=0.0.0a0
- &cudf_unsuffixed cudf==24.8.*,>=0.0.0a0
- output_types: requirements
packages:
# pip recognizes the index as a global option for the requirements.txt file
Expand All @@ -716,18 +737,22 @@ dependencies:
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
- matrix:
cuda: "12.*"
cuda_suffixed: "true"
packages:
- cudf-cu12==24.8.*,>=0.0.0a0
- matrix: {cuda: "11.*"}
- matrix:
cuda: "11.*"
cuda_suffixed: "true"
packages:
- cudf-cu11==24.8.*,>=0.0.0a0
- {matrix: null, packages: [*cudf_conda]}
- {matrix: null, packages: [*cudf_unsuffixed]}
depends_on_cudf_kafka:
common:
- output_types: conda
packages:
- &cudf_kafka_conda cudf_kafka==24.8.*,>=0.0.0a0
- &cudf_kafka_unsuffixed cudf_kafka==24.8.*,>=0.0.0a0
- output_types: requirements
packages:
# pip recognizes the index as a global option for the requirements.txt file
Expand All @@ -737,13 +762,17 @@ dependencies:
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix: {cuda: "12.*"}
- matrix:
cuda: "12.*"
cuda_suffixed: "true"
packages:
- cudf_kafka-cu12==24.8.*,>=0.0.0a0
- matrix: {cuda: "11.*"}
- matrix:
cuda: "11.*"
cuda_suffixed: "true"
packages:
- cudf_kafka-cu11==24.8.*,>=0.0.0a0
- {matrix: null, packages: [*cudf_kafka_conda]}
- {matrix: null, packages: [*cudf_kafka_unsuffixed]}
depends_on_cupy:
common:
- output_types: conda
Expand Down
1 change: 1 addition & 0 deletions python/cudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ skip = [
[tool.rapids-build-backend]
build-backend = "scikit_build_core.build"
dependencies-file = "../../dependencies.yaml"
matrix-entry = "cuda_suffixed=true"
requires = [
"cmake>=3.26.4,!=3.30.0",
"cython>=3.0.3",
Expand Down
1 change: 1 addition & 0 deletions python/cudf_kafka/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ regex = "(?P<value>.*)"
[tool.rapids-build-backend]
build-backend = "scikit_build_core.build"
dependencies-file = "../../dependencies.yaml"
matrix-entry = "cuda_suffixed=true"
requires = [
"cmake>=3.26.4,!=3.30.0",
"cython>=3.0.3",
Expand Down
1 change: 1 addition & 0 deletions python/cudf_polars/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,3 +182,4 @@ docstring-code-format = true
[tool.rapids-build-backend]
build-backend = "setuptools.build_meta"
dependencies-file = "../../dependencies.yaml"
matrix-entry = "cuda_suffixed=true"
1 change: 1 addition & 0 deletions python/custreamz/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Homepage = "https://github.com/rapidsai/cudf"
[tool.rapids-build-backend]
build-backend = "setuptools.build_meta"
dependencies-file = "../../dependencies.yaml"
matrix-entry = "cuda_suffixed=true"

[tool.setuptools]
license-files = ["LICENSE"]
Expand Down
1 change: 1 addition & 0 deletions python/dask_cudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Homepage = "https://github.com/rapidsai/cudf"
[tool.rapids-build-backend]
build-backend = "setuptools.build_meta"
dependencies-file = "../../dependencies.yaml"
matrix-entry = "cuda_suffixed=true"

[tool.setuptools]
license-files = ["LICENSE"]
Expand Down
Loading