Update dependency.yaml fallback entries to (unsuffixed) CUDA 12 dependencies

[jameslamb]

Description

Created from @bdice's suggestion at rapidsai/cudf#15245 (comment)

RAPIDS projects' dependencies.yaml files should be modified such that the no-CUDA-version lists contain the set of dependencies needed when targeting CUDA 12 (but without suffixes like -cu12).

For example, consider the following from cudf

run_cudf:
  # ...
  specific:
    - output_types: [conda, requirements, pyproject]
      matrices:
        - matrix: {cuda: "12.*"}
          packages:
            - cuda-python>=12.0,<13.0a0
        - matrix: {cuda: "11.*"}
          packages: &run_cudf_packages_all_cu11
            - cuda-python>=11.7.1,<12.0a0
        - {matrix: null, packages: *run_cudf_packages_all_cu11}
    - output_types: conda
      matrices:
        - matrix: {cuda: "12.*"}
          packages:
            - pynvjitlink
        - matrix: {cuda: "11.*"}
          packages:
            - cubinlinker
            - ptxcompiler
    - output_types: [requirements, pyproject]
      matrices:
        - matrix: {cuda: "12.*"}
          packages:
            - pynvjitlink-cu12
        - matrix: {cuda: "11.*"}
          packages:
            - cubinlinker-cu11
            - ptxcompiler-cu11
        - {matrix: null, packages: [cubinlinker, ptxcompiler]}

If you run rapids-dependency-file-generator --output requirements against that without a cuda value provided, it'll produce something like this.

cuda-python>=11.7.1,<12.0a0
cubinlinker
ptxcompiler

Those fallback dependencies.yaml blocks should be modified such that it instead generates something like this.

cuda-python>=12.0,<13.0a0
pynvjitlink

With a change like this:

--- before.txt	2024-05-29 14:58:15
+++ after.txt	2024-05-29 14:58:27
@@ -4,12 +4,12 @@
     - output_types: [conda, requirements, pyproject]
       matrices:
         - matrix: {cuda: "12.*"}
-          packages:
+          packages: &run_cudf_package_all_cu12
             - cuda-python>=12.0,<13.0a0
         - matrix: {cuda: "11.*"}
-          packages: &run_cudf_packages_all_cu11
+          packages:
             - cuda-python>=11.7.1,<12.0a0
-        - {matrix: null, packages: *run_cudf_packages_all_cu11}
+        - {matrix: null, packages: *run_cudf_packages_all_cu12}
     - output_types: conda
       matrices:
         - matrix: {cuda: "12.*"}
@@ -28,5 +28,5 @@
           packages:
             - cubinlinker-cu11
             - ptxcompiler-cu11
-        - {matrix: null, packages: [cubinlinker, ptxcompiler]}
+        - {matrix: null, packages: [pynvjitlink]}

Benefits of this work

• pyproject.toml / requirements*.txt checked in source control will look more like the dependencies for building the library against the latest supported version of CUDA
  • ref: https://github.com/rapidsai/cuml/pull/5804/files#r1586342273
• for DLFW builds, which are often ahead of RAPIDS on CUDA version (and which use these fallback matrices to avoid depending on suffixed stuff like cudf-cu12), removes some need to patch requirements files

Acceptance Criteria

For every project:

• running rapids-dependency-file-generator --output requirements or --output pyproject for any keys that produce those outputs produces a list of dependencies for CUDA 12 (without suffixes like -cu12)

Approach

Some projects might need 0 changes... this situation where projects depend on different libraries (not just different versions / suffixed names) across CUDA major versions isn't universal.

For the others that do need changes, modify them following the cudf example from above.

Notes

Isn't rapids-build-backend supposed to solve this?

As of today, it doesn't.

It supports an option, disable-cuda, which if true is interpreted as "do not try to guess the target CUDA version... leave project name and dependency names unsuffixed, and use the fallback matrices in dependencies.yaml"

(rapids-build-backend-docs.

Those things could be decomposed into separate settings, so that you could for example do something like this with a project using rapids-build-backend :

pip wheel \
    -C rapidsai.update-project-cuda-suffix=false \
    -C rapidsai.update-dependency-suffixes=false \
    -C rapidsai.target-cuda-version="12.2.2" \
    .

(or set any of those via environment variables instead of -C flags passed to the builder)

Related conversation: rapidsai/cudf#15245 (comment).

But in the current state of rapids-build-backend, that isn't supported.

Since the dependencies.yaml updates should be low-effort and wouldn't preclude something like the rapids-build-backend API changes, and since they have at least one other benefit (pyproject.toml / requirements.txt in source control more closely matching what'll be pulled in when targeting the latest CUDA version) I think it'd be worth making these dependencies.yaml changes regardless of any decision about how to modify rapids-build-backend to support DLFW-style builds.

[jameslamb]

@bdice @trxcllnt @KyleFromNVIDIA I know we reached a tentative conclusion on a call earlier today that this should be unnecessary, but I think it'd still be useful.

I wrote this up to hopefully clarify the exact design question(s) and what rapids-build-backend currently can / cannot do.