cross compile cuda support (cnt'd) #210

h-vetinari · 2022-11-04T02:21:55Z

New PR since I cannot commit into #209

Closes #209

conda-forge-linter · 2022-11-04T02:21:59Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

recipe/cross_compile_support.sh

h-vetinari · 2022-11-04T03:42:36Z

Thanks for the quick review @kkraus14! Do you know whether we need the other devel packages?

kkraus14 · 2022-11-04T04:02:21Z

Thanks for the quick review @kkraus14! Do you know whether we need the other devel packages?

Yes we'll need those. Those contain the headers, the generic unversioned symlinked shared library, the static libraries that we may want specifically for CUDA because there's some implications IIRC, and the pkgconfig files.

h-vetinari · 2022-11-04T04:16:00Z

Argh, the test here cannot work because at test time, the "BUILD_PLATFORM" is already emulated.

Yes we'll need those.

Added in 91610af; not sure if my scripting is worth a damn, I'm not experienced with jq, but I tested it in https://jqplay.org/ at least, and of course bash errors are a real possibility too. 😅

kkraus14 · 2022-11-04T04:14:06Z

recipe/cross_compile_support.sh

+                # (names need "_" not "-" to match spelling in manifest)
+                declare -a DEVELS=(
+                    "cuda_cudart"
+                    "cuda_driver"


I believe you need to use the cuda_cudart version for cuda_driver since it doesn't have its own key.

kkraus14 · 2022-11-04T04:39:15Z

recipe/cross_compile_support.sh

+                declare -a DEVELS=(
+                    "cuda_cudart"
+                    "cuda_driver"
+                    "cuda_nvml"
+                    "cuda_nvrtc"
+                    "libcublas"
+                    "libcufft"
+                    "libcurand"
+                    "libcusolver"
+                    "libcusparse"
+                    "libnpp"
+                    "libnvjpeg"
+                )


Need to add: nvidia_driver, nvidia_fabric_manager, nvidia_libXNVCtrl. All three should use the version from the nvidia_driver key

OK, this and the cuda_driver thing make it sound like we need to do a full mapping here. Currently it was just a simple scheme: xyz -> xyz_devel.

You're saying I need to change this to:

# new key... ...uses version from cuda_cudart_devel -> cuda_cudart cuda_driver_devel -> cuda_cudart # special cuda_nvrtc_devel -> cuda_nvrtc cuda_nvml_devel -> cuda_nvml_dev # special libcublas_devel -> libcublas libcufft_devel -> libcufft libcurand_devel -> libcurand libcusolver_devel -> libcusolver libcusparse_devel -> libcusparse libnpp_devel -> libnpp libnvjpeg_devel -> libnvjpeg nvidia_driver_devel -> nvidia_driver nvidia_fabric_manager -> nvidia_driver # special nvidia_libXNVCtrl -> nvidia_driver # special

Those are not needed.

Can you specify what you mean by "those"? The original list was based on the devel-libs you had in your PR, but weren't listed in the manifest

nvidia_libXNVCtrl is GPL licensed (https://github.com/NVIDIA/nvidia-settings/blob/main/COPYING) and not governed under the EULA so I think (I am not a lawyer!) we can actually ship that as a conda package if desired so that can definitely be removed from the list in hindsight. My apologies.

I can't find a license or EULA for the nvidia_fabric_manager (should actually have been nvidia_fabric_manager_devel, my bad again), so I assumed we weren't allowed to ship it as a conda package. I haven't seen anyone actually use this library before so I'd be perfectly happy to leave it out.

There's other things that would potentially be needed that there aren't stub libraries available for:

nvidia-driver-libs notably contains libnvoptix and a bunch of other libraries

nvidia-driver-cuda-libs notably contains libnvidia-nvvm and libnvidia-ptxjitcompiler and more generally contains the libraries that the symlinks in nvidia_driver_devel point to

I have seen a fair amount of software that uses these options where I think it would make sense to add these libraries: nvidia_driver_devel, nvidia_driver_libs, and nvidia_driver_cuda_libs.

Notably with the proposed above, we'll be reliant on stubs for the CUDA driver library libcuda and the NVIDIA Management Library libnvidia-ml. I'm not sure how build systems like CMake will react to using a mix of actual and stub libraries. Hopefully everything will be okay?

nvidia-libXNVCtrl{,-devel} and nvidia-fabric-manager{,-devel} don't exist under:
https://developer.download.nvidia.com/compute/cuda/repos/rhel8/cross-linux-sbsa/
https://developer.download.nvidia.com/compute/cuda/repos/rhel8/ppc64le/

recipe/cross_compile_support.sh

isuruf

This is way more complicated and downloads more things than needed. The packages I had in my PR were the only ones in $CUDA_HOME/targets/sbsa-linux which is what the cross compiler looks at. All the other libraries are irrelevant.

h-vetinari · 2022-11-04T21:02:37Z

This is way more complicated and downloads more things than needed.

Fair enough, I was just following the instructions to download packages based on the manifest. I started with downloading everything (+ what Keith told me to add), but if we can filter/shorten the list, all the better

kkraus14 · 2022-11-05T03:57:43Z

This is way more complicated and downloads more things than needed. The packages I had in my PR were the only ones in $CUDA_HOME/targets/sbsa-linux which is what the cross compiler looks at. All the other libraries are irrelevant.

I think there's a handful of other libraries that end up in the sysroot (I think?) that there aren't stubs for that we'd want as well that weren't included in your PR. I did my best to break them down in my comment in the thread.

h-vetinari · 2022-11-05T06:02:57Z

@isuruf: This is way more complicated and downloads more things than needed.

@kkraus14: There's other things that would potentially be needed [...]

I'll let you two decide what should be included.

My goal here was just to make it possible to use the versions from the manifest. As feared, this turned into a fair amount of code, due to various small and not so small divergences between the manifest and the actual RPMs. It's conceivable that someone might prefer to just hardcode the list of deps, but now that it works (w/ ability to map in additional rpms, and the obvious possibility to filter out unwanted packages), I tend to think it's probably a good thing going forward.

@isuruf, I managed to get this to run through to the point where you had added

mv ./usr/local/cuda-${CUDA_COMPILER_VERSION}/targets/${CUDA_HOST_PLATFORM_ARCH}-linux ${CUDA_HOME}/targets/${CUDA_HOST_PLATFORM_ARCH}-linux

at the end of the loop, but this fails with

mv: cannot move ‘./usr/local/cuda-11.2/targets/sbsa-linux’ to ‘/usr/local/cuda/targets/sbsa-linux’: Permission denied

h-vetinari · 2022-11-06T23:31:05Z

@isuruf @kkraus14
Can we iterate on the list of things to install?

At the moment the list is as follows:

{
  "cuda": {
    "name": "CUDA SDK",
    "version": "11.2.2"
  },
  "cuda_cudart": {
    "name": "CUDA Runtime (cudart)",
    "version": "11.2.152"
  },
  "cuda_cuobjdump": {
    "name": "cuobjdump",
    "version": "11.2.152"
  },
  "cuda_cupti": {
    "name": "CUPTI",
    "version": "11.2.152"
  },
  "cuda_cuxxfilt": {
    "name": "CUDA cu++ filt",
    "version": "11.2.152"
  },
  "cuda_gdb": {
    "name": "CUDA GDB",
    "version": "11.2.152"
  },
  "cuda_nvcc": {
    "name": "CUDA NVCC",
    "version": "11.2.152"
  },
  "cuda_nvdisasm": {
    "name": "CUDA nvdisasm",
    "version": "11.2.152"
  },
  "cuda_nvml_dev": {
    "name": "CUDA NVML Headers",
    "version": "11.2.152"
  },
  "cuda_nvprof": {
    "name": "CUDA nvprof",
    "version": "11.2.152"
  },
  "cuda_nvprune": {
    "name": "CUDA nvprune",
    "version": "11.2.152"
  },
  "cuda_nvrtc": {
    "name": "CUDA NVRTC",
    "version": "11.2.152"
  },
  "cuda_nvtx": {
    "name": "CUDA NVTX",
    "version": "11.2.152"
  },
  "cuda_samples": {
    "name": "CUDA Samples",
    "version": "11.2.152"
  },
  "cuda_sanitizer_api": {
    "name": "CUDA Compute Sanitizer API",
    "version": "11.2.152"
  },
  "libcublas": {
    "name": "CUDA cuBLAS",
    "version": "11.4.1.1043"
  },
  "libcufft": {
    "name": "CUDA cuFFT",
    "version": "10.4.1.152"
  },
  "libcurand": {
    "name": "CUDA cuRAND",
    "version": "10.2.3.152"
  },
  "libcusolver": {
    "name": "CUDA cuSOLVER",
    "version": "11.1.0.152"
  },
  "libcusparse": {
    "name": "CUDA cuSPARSE",
    "version": "11.4.1.1152"
  },
  "libnpp": {
    "name": "CUDA NPP",
    "version": "11.3.2.152"
  },
  "libnvjpeg": {
    "name": "CUDA nvJPEG",
    "version": "11.4.0.152"
  },
  "nsight_compute": {
    "name": "Nsight Compute",
    "version": "2020.3.1.4"
  },
  "nsight_systems": {
    "name": "Nsight Systems",
    "version": "2020.4.3.7"
  },
  "nvidia_driver": {
    "name": "NVIDIA Linux Driver",
    "version": "460.32.03"
  },
  # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  # from manifest up until this point;
  # entries below added by mapping
  # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
  "cuda_cudart_devel": {
    "version": "11.2.152"
  },
  "cuda_driver_devel": {
    "version": "11.2.152"
  },
  "cuda_nvrtc_devel": {
    "version": "11.2.152"
  },
  "libcublas_devel": {
    "version": "11.4.1.1043"
  },
  "libcufft_devel": {
    "version": "10.4.1.152"
  },
  "libcurand_devel": {
    "version": "10.2.3.152"
  },
  "libcusolver_devel": {
    "version": "11.1.0.152"
  },
  "libcusparse_devel": {
    "version": "11.4.1.1152"
  },
  "libnpp_devel": {
    "version": "11.3.2.152"
  },
  "libnvjpeg_devel": {
    "version": "11.4.0.152"
  },
  "nvidia_driver_devel": {
    "version": "460.32.03"
  }
}

h-vetinari · 2022-11-07T03:21:10Z

More compact list:

cuda:11.2.2
cuda-cudart:11.2.152
cuda-cudart-devel:11.2.152
cuda-cuobjdump:11.2.152
cuda-cupti:11.2.152
cuda-cuxxfilt:11.2.152
cuda-driver-devel:11.2.152
cuda-gdb:11.2.152
cuda-nvcc:11.2.152
cuda-nvdisasm:11.2.152
cuda-nvml-devel:11.2.152
cuda-nvprof:11.2.152
cuda-nvprune:11.2.152
cuda-nvrtc:11.2.152
cuda-nvrtc-devel:11.2.152
cuda-nvtx:11.2.152
cuda-samples:11.2.152
cuda-sanitizer:11.2.152
libcublas:11.4.1.1043
libcublas-devel:11.4.1.1043
libcufft:10.4.1.152
libcufft-devel:10.4.1.152
libcurand:10.2.3.152
libcurand-devel:10.2.3.152
libcusolver:11.1.0.152
libcusolver-devel:11.1.0.152
libcusparse:11.4.1.1152
libcusparse-devel:11.4.1.1152
libnpp:11.3.2.152
libnpp-devel:11.3.2.152
libnvjpeg:11.4.0.152
libnvjpeg-devel:11.4.0.152
nsight-compute:2020.3.1.4
nsight-systems:2020.4.3.7
nvidia-driver:460.32.03
nvidia-driver-devel:460.32.03

h-vetinari · 2022-11-07T03:22:57Z

@isuruf, I managed to get this to run through to the point where you had added
mv ./usr/local/cuda-${CUDA_COMPILER_VERSION}/targets/${CUDA_HOST_PLATFORM_ARCH}-linux ${CUDA_HOME}/targets/${CUDA_HOST_PLATFORM_ARCH}-linux
at the end of the loop, but this fails with
mv: cannot move ‘./usr/local/cuda-11.2/targets/sbsa-linux’ to ‘/usr/local/cuda/targets/sbsa-linux’: Permission denied

I'm pretty sure now that this happens because I was running this in the recipe-section for debugging purposes, where we don't actually have root rights. I think that if this script is run where it should, that actually should run through. In any case, that it works up until this point shows that the downloads are fine, which means I'm marking this as ready (happy to clean up commit history if desired).

h-vetinari · 2022-11-07T03:34:49Z

.ci_support/migrations/cuda_112_ppc64le_aarch64.yaml

+docker_image:                                           # [os.environ.get("BUILD_PLATFORM", "").startswith("linux") and (ppc64le or aarch64)]
+   # case: native compilation (build == target)
+   - quay.io/condaforge/linux-anvil-ppc64le-cuda:11.2   # [ppc64le and os.environ.get("BUILD_PLATFORM") == "linux-ppc64le"]
+   - quay.io/condaforge/linux-anvil-aarch64-cuda:11.2   # [aarch64 and os.environ.get("BUILD_PLATFORM") == "linux-aarch64"]
+   # case: cross-compilation (build != target)
+   - quay.io/condaforge/linux-anvil-cuda:11.2           # [ppc64le and os.environ.get("BUILD_PLATFORM") == "linux-64"]
+   - quay.io/condaforge/linux-anvil-cuda:11.2           # [aarch64 and os.environ.get("BUILD_PLATFORM") == "linux-64"]


Not sure if adding this migration is strictly speaking necessary, but since the linux builds have both cuda / non-cuda, I think we should do the same for aarch/ppc?

This reflects the changes in conda-forge/conda-forge-pinning-feedstock#3624, which we should ideally merge before-hand (then we can skip the use_local: true here).

Please remove this until nvcc-feedstock is ready

No problem, I'll remove it; though I wonder if we'll still be able to cross-compile cuda without this (which is the point of this PR, no?)

How do you propose we solve the issue in #210 (comment)?

How do you propose we solve the issue in #210 (comment)?

I don't know yet, I'm trying to help work through the issues as they appear. But the overarching goal remains to cross-compile CUDA, so if we're falling short of that, we should keep this open IMO.

h-vetinari · 2022-11-07T03:41:29Z

I'm pretty sure now that this happens because I was running this in the recipe-section for debugging purposes, where we don't actually have root rights. I think that if this script is run where it should, that actually should run through.

This turned out to be wrong... So now I'm not sure what we need to tweak (chown?) to get the packages into $CUDA_HOME. It's a "simple" permission issue, but I don't know well enough which layer here is allowed to do what.

Also worth noting - not everything gets installed in ./usr/local/cuda-11.2, some goes in ./opt/nvidia or ./usr/share.

with changes from conda-forge/conda-forge-pinning-feedstock#3624

…nda-forge-pinning 2023.01.30.13.41.09

This reverts commit f0ba50f.

…nda-forge-pinning 2023.02.01.21.26.02

h-vetinari · 2023-02-02T01:32:51Z

The committer being you is fine. I just don't want my commit squashed.

OK, I'll keep this in mind. For context, I often need to trawl through feedstock history, and I find sequences of commits like:

really annoying (aside from the content-free commit messages that the GH UI encourages), because they tackle one specific issue and semantically should be grouped together. This is not a dig at your commits here, I do the same during development as well, I just have the habit of cleaning up my commit history as I go resp. before merging, so that it's easier to navigate down the line.

But now that I know that you don't like that, I'll make an exception for your commits.

isuruf · 2023-02-02T01:34:30Z

If you want, you can clean up the commit message when there's no message, but keep my commits as they are in the future.

h-vetinari · 2023-02-02T01:43:53Z

@isuruf: How do you propose we solve the issue in #210 (comment)?

@h-vetinari: I don't know yet, I'm trying to help work through the issues as they appear. But the overarching goal remains to cross-compile CUDA, so if we're falling short of that, we should keep this open IMO.

So what are the next steps we need to tackle for cross-compiling CUDA, and where do we track them (if not here)?

isuruf · 2023-02-02T01:56:21Z

We can track them in nvcc-feedstock

recipe/cross_compile_support.sh

h-vetinari changed the title ~~cross compile cuda support~~ cross compile cuda support (cnt'd) Nov 4, 2022

h-vetinari mentioned this pull request Nov 4, 2022

cross compile cuda support #209

Closed

5 tasks

kkraus14 reviewed Nov 4, 2022

View reviewed changes

recipe/cross_compile_support.sh Outdated Show resolved Hide resolved

recipe/cross_compile_support.sh Show resolved Hide resolved

kkraus14 reviewed Nov 4, 2022

View reviewed changes

h-vetinari force-pushed the cross branch 2 times, most recently from fbb536a to ca40910 Compare November 4, 2022 06:06

h-vetinari commented Nov 4, 2022

View reviewed changes

recipe/cross_compile_support.sh Outdated Show resolved Hide resolved

h-vetinari commented Nov 4, 2022

View reviewed changes

recipe/cross_compile_support.sh Outdated Show resolved Hide resolved

isuruf reviewed Nov 4, 2022

View reviewed changes

h-vetinari force-pushed the cross branch from 850b3cf to e9c36ca Compare November 5, 2022 04:47

h-vetinari force-pushed the cross branch from 3643e95 to 81a72e8 Compare November 7, 2022 03:11

h-vetinari marked this pull request as ready for review November 7, 2022 03:23

h-vetinari requested a review from a team as a code owner November 7, 2022 03:23

h-vetinari force-pushed the cross branch from 79e8b44 to 4cba359 Compare November 7, 2022 03:29

h-vetinari commented Nov 7, 2022

View reviewed changes

h-vetinari mentioned this pull request Nov 7, 2022

arrow-cpp v10.0.1 conda-forge/arrow-cpp-feedstock#875

Merged

3 tasks

h-vetinari force-pushed the cross branch 3 times, most recently from f8078db to 7b2c42b Compare November 8, 2022 03:36

h-vetinari and others added 11 commits February 1, 2023 19:32

add cross-compatible cuda migrator for aarch/ppc

352608b

with changes from conda-forge/conda-forge-pinning-feedstock#3624

cosmetics

4e42010

filter packages from manifest down to what we need for cross-compilation

2c7c921

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

e67e240

…nda-forge-pinning 2023.01.30.13.41.09

copy through /opt/conda/target to pass sudoers rule

d2a84c9

also try to create ${CUDA_HOME}/targets/${CUDA_HOST_PLATFORM_ARCH}-linux

7183e62

Use sudo

9144456

Update cross_compile_support.sh

5327ca3

Update cross_compile_support.sh

0ac13de

Revert "add cross-compatible cuda migrator for aarch/ppc"

3226d52

This reverts commit f0ba50f.

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

1391240

…nda-forge-pinning 2023.02.01.21.26.02

isuruf force-pushed the cross branch from bb3af84 to 1391240 Compare February 2, 2023 01:33

isuruf added 2 commits February 1, 2023 19:35

Need libarchive for bsdtar command

103d35d

Update version

1b00d7c

Don't set -e in files that are sourced

f47e98c

isuruf requested a review from kkraus14 February 2, 2023 02:21

kkraus14 reviewed Feb 2, 2023

View reviewed changes

recipe/cross_compile_support.sh Show resolved Hide resolved

recipe/cross_compile_support.sh Show resolved Hide resolved

kkraus14 approved these changes Feb 2, 2023

View reviewed changes

isuruf merged commit d1436f4 into conda-forge:main Feb 2, 2023

h-vetinari deleted the cross branch February 3, 2023 00:17

h-vetinari mentioned this pull request Feb 24, 2023

Arrow: Support CUDA 12 conda-forge/arrow-cpp-feedstock#974

Closed

This was referenced Apr 14, 2023

expand cuda_112_ppc64le_aarch64 migrator for cross-compilation conda-forge/conda-forge-pinning-feedstock#3624

Merged

Cross-compiling CUDA: next steps conda-forge/nvcc-feedstock#95

Closed

fix typo #235

Merged

h-vetinari mentioned this pull request Jul 14, 2023

cross-compiled CUDA builds running out of disk space conda-forge/arrow-cpp-feedstock#1114

Closed

h-vetinari mentioned this pull request Aug 23, 2023

Add CUDA 11.8 cross-compilation support #261

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cross compile cuda support (cnt'd) #210

cross compile cuda support (cnt'd) #210

h-vetinari commented Nov 4, 2022 •

edited

Loading

conda-forge-linter commented Nov 4, 2022

h-vetinari commented Nov 4, 2022

kkraus14 commented Nov 4, 2022

h-vetinari commented Nov 4, 2022

kkraus14 Nov 4, 2022

kkraus14 Nov 4, 2022

h-vetinari Nov 4, 2022 •

edited

Loading

isuruf Nov 4, 2022

h-vetinari Nov 4, 2022

kkraus14 Nov 5, 2022

h-vetinari Nov 5, 2022

isuruf left a comment •

edited

Loading

h-vetinari commented Nov 4, 2022

kkraus14 commented Nov 5, 2022

h-vetinari commented Nov 5, 2022 •

edited

Loading

h-vetinari commented Nov 6, 2022

h-vetinari commented Nov 7, 2022

h-vetinari commented Nov 7, 2022

h-vetinari Nov 7, 2022

isuruf Feb 2, 2023

h-vetinari Feb 2, 2023

isuruf Feb 2, 2023 •

edited

Loading

h-vetinari Feb 2, 2023

h-vetinari commented Nov 7, 2022

h-vetinari commented Feb 2, 2023 •

edited

Loading

isuruf commented Feb 2, 2023

h-vetinari commented Feb 2, 2023

isuruf commented Feb 2, 2023

cross compile cuda support (cnt'd) #210

cross compile cuda support (cnt'd) #210

Conversation

h-vetinari commented Nov 4, 2022 • edited Loading

conda-forge-linter commented Nov 4, 2022

h-vetinari commented Nov 4, 2022

kkraus14 commented Nov 4, 2022

h-vetinari commented Nov 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isuruf left a comment • edited Loading

Choose a reason for hiding this comment

h-vetinari commented Nov 4, 2022

kkraus14 commented Nov 5, 2022

h-vetinari commented Nov 5, 2022 • edited Loading

h-vetinari commented Nov 6, 2022

h-vetinari commented Nov 7, 2022

h-vetinari commented Nov 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isuruf Feb 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari commented Nov 7, 2022

h-vetinari commented Feb 2, 2023 • edited Loading

isuruf commented Feb 2, 2023

h-vetinari commented Feb 2, 2023

isuruf commented Feb 2, 2023

h-vetinari commented Nov 4, 2022 •

edited

Loading

h-vetinari Nov 4, 2022 •

edited

Loading

isuruf left a comment •

edited

Loading

h-vetinari commented Nov 5, 2022 •

edited

Loading

isuruf Feb 2, 2023 •

edited

Loading

h-vetinari commented Feb 2, 2023 •

edited

Loading