Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add libcufile recipe #21908

Merged
merged 72 commits into from
Apr 12, 2023
Merged
Show file tree
Hide file tree
Changes from 61 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
b5e65d6
Add libcufile recipe
mroeschke Jan 31, 2023
c9f1fc1
Add Licence metadata
mroeschke Jan 31, 2023
372f26e
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
488dbda
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
4279763
Add summary
mroeschke Jan 31, 2023
1a4969e
Change description to summary
mroeschke Jan 31, 2023
0c20a35
Add file tests
mroeschke Jan 31, 2023
fbac483
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
e41a0ac
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
5d18921
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
cdbcdba
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
9e6d832
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
ee146b6
Add build.sh instead of build.bat
mroeschke Feb 1, 2023
9ad326f
Update recipes/libcufile/Makefile.patch
mroeschke Feb 3, 2023
64b3393
some formatting
mroeschke Feb 3, 2023
a90df31
Try patchfile fix
mroeschke Feb 4, 2023
702a290
Add directory
mroeschke Feb 4, 2023
dbbd113
Create and move tools and samples to gds dir
mroeschke Feb 7, 2023
8e21b79
Merge branch 'main' into libcufile
mroeschke Feb 7, 2023
1970a53
typo -p
mroeschke Feb 7, 2023
6158a96
just -d
mroeschke Feb 7, 2023
7497aed
Move man
mroeschke Feb 7, 2023
c8e9e42
include new gds dir
mroeschke Feb 7, 2023
a17812a
Update recipes/libcufile/meta.yaml
mroeschke Feb 8, 2023
5317c44
Move to difference section
mroeschke Feb 8, 2023
f264627
Move to outputs build
mroeschke Feb 8, 2023
f93c0b2
New build section
mroeschke Feb 8, 2023
11ee0dc
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
79c750a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
c8de857
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
54a340a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
178099f
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
1ba2afe
Update recipes/libcufile/meta.yaml
mroeschke Feb 11, 2023
76cd37c
Merge branch 'main' into libcufile
mroeschke Feb 11, 2023
61ab85d
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
93593ce
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
498ac3a
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
89875b4
wildcards missing?
mroeschke Feb 14, 2023
86b9763
Revert "wildcards missing?"
mroeschke Feb 16, 2023
7939393
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9cd3345
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
2556d27
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9d4601f
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
0452766
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
28723e1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
f03b6c1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
be9e78c
Merge branch 'main' into libcufile
mroeschke Mar 22, 2023
900a8aa
Merge branch 'main' into libcufile
jakirkham Mar 27, 2023
1ed22aa
Merge branch 'main' into libcufile
mroeschke Apr 5, 2023
25ee3e5
Make changes to meta and build files
mroeschke Apr 5, 2023
c671519
Keep top level about section
mroeschke Apr 5, 2023
afa2ad6
Merge branch 'main' into libcufile
mroeschke Apr 7, 2023
ba1e05d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
ca53dd3
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
21f096d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
694ac62
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
32b88d0
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
3cda9c8
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
23d673c
Refactor -dev tests
mroeschke Apr 7, 2023
ebfbf07
pin_compatible?
mroeschke Apr 7, 2023
f32c306
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
7456ba7
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
296b975
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
08a0182
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
4b402b1
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
9e35a54
Fix libcufile tests and packaging.
bdice Apr 12, 2023
c1f3c4b
Attempt to fix RDMA issue.
bdice Apr 12, 2023
5b28f1b
Make tests pass.
bdice Apr 12, 2023
08a70e1
Fix RDMA cdt dependency.
bdice Apr 12, 2023
e1d5f02
Don't ship etc/cufile.json.
bdice Apr 12, 2023
16cfcdd
Rerun CI.
bdice Apr 12, 2023
21053dd
Merge pull request #1 from bdice/fix-libcufile-tests
mroeschke Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions recipes/libcufile/Makefile.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
--- samples/Makefile 2022-06-24 00:23:23.000000000 +0000
+++ samples/Makefile.new 2022-06-27 15:04:32.235125647 +0000
@@ -35,8 +35,8 @@

# Common includes and paths for CUDA
-CUDA_PATH := /usr/local/cuda
-CUFILE_PATH ?= /usr/local/cuda/targets/x86_64-linux/lib/
-CUFILE_INCLUDE_PATH ?= /usr/local/cuda/targets/x86_64-linux/include/
+CUDA_PATH := ${PREFIX}
+CUFILE_PATH ?= $(CUDA_PATH)/targets/x86_64-linux/lib/
+CUFILE_INCLUDE_PATH ?= $(CUDA_PATH)/targets/x86_64-linux/include/
CXXFLAGS := -Wall
CXXFLAGS += -I $(CUDA_PATH)/include/
CXXFLAGS += -I $(CUFILE_INCLUDE_PATH)

34 changes: 34 additions & 0 deletions recipes/libcufile/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash

# Install to conda style directories
[[ -d lib64 ]] && mv lib64 lib
mkdir -p ${PREFIX}/lib
mkdir -p ${PREFIX}/gds
mv samples ${PREFIX}/gds/samples
mv tools ${PREFIX}/gds/tools
[[ -d pkg-config ]] && mv pkg-config ${PREFIX}/lib/pkgconfig
[[ -d "$PREFIX/lib/pkgconfig" ]] && sed -E -i "s|cudaroot=.+|cudaroot=$PREFIX|g" $PREFIX/lib/pkgconfig/cufile*.pc

[[ ${target_platform} == "linux-64" ]] && targetsDir="targets/x86_64-linux"
[[ ${target_platform} == "linux-aarch64" ]] && targetsDir="targets/sbsa-linux"

for i in `ls`; do
[[ $i == "build_env_setup.sh" ]] && continue
[[ $i == "conda_build.sh" ]] && continue
[[ $i == "metadata_conda_debug.yaml" ]] && continue
if [[ $i == "lib" ]] || [[ $i == "gds" ]] || [[ $i == "include" ]]; then
mkdir -p ${PREFIX}/${targetsDir}
mkdir -p ${PREFIX}/$i
cp -rv $i ${PREFIX}/${targetsDir}
if [[ $i == "lib" ]]; then
for j in "$i"/*.so*; do
# Shared libraries are symlinked in $PREFIX/lib
ln -s ${PREFIX}/${targetsDir}/$j ${PREFIX}/$j
done
fi
else
# Put all other files in targetsDir
mkdir -p ${PREFIX}/${targetsDir}/${PKG_NAME}
cp -rv $i ${PREFIX}/${targetsDir}/${PKG_NAME}
fi
done
2 changes: 2 additions & 0 deletions recipes/libcufile/conda_build_config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
arm_variant_type: # [aarch64]
- sbsa # [aarch64]
157 changes: 157 additions & 0 deletions recipes/libcufile/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
{% set name = "libcufile" %}
{% set version = "1.5.0.59" %}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
{% set cuda_version = "12.0" %}
{% set platform = "linux-x86_64" %} # [linux64]
{% set platform = "linux-sbsa" %} # [aarch64]
{% set target_name = "x86_64-linux" %} # [linux64]
{% set target_name = "sbsa-linux" %} # [aarch64]
{% set extension = "tar.xz" %} # [not win]

package:
name: {{ name|lower }}
version: {{ version }}

source:
url: https://developer.download.nvidia.com/compute/cuda/redist/{{ name }}/{{ platform }}/{{ name }}-{{ platform }}-{{ version }}-archive.{{ extension }}
sha256: 222c49b29afb544c69542c4e658b0a1bd01fd2306f2136ad6990ccfff482a097 # [linux64]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
sha256: 4e8d2b525322be9ab6ae574bd1d0325a4bf9c6c3bbdaa1611b5b969b7b34b0c4 # [aarch64]
patches:
- Makefile.patch

build:
number: 0
skip: true # [not linux]

outputs:
- name: libcufile
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
build:
missing_dso_whitelist:
- "*/libcuda.so.*" # [linux]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
run_exports:
- {{ pin_subpackage("libcufile", max_pin="x") }}
files:
- lib/libcufile*.so.*
- etc/cufile.json
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's already been checked, but does cuFile use the file put here as expected?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good question. Will ask

It's possible we would need to set CUFILE_ENV_PATH_JSON for this to work. That could be done with activation script

Suppose a fair question would be whether we want to do this or whether users would rather maintain their own configuration file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we don't need this file. So we dropped it to avoid confusion

We can always revisit this in the feedstock

- man
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
test:
commands:
- test -f $PREFIX/lib/libcufile.so.*
- test -f $PREFIX/lib/libcufile_rdma.so.*
- test -f $PREFIX/etc/cufile.json
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: libcufile-dev
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
build:
run_exports:
- {{ pin_subpackage("libcufile", max_pin="x") }}
files:
- include/cufile*
- lib/libcufile*.so
- targets/{{ target_name }}/lib/libcufile*.so
- targets/{{ target_name }}/include/cufile*
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
requirements:
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
- {{ pin_subpackage("libcufile", exact=True) }}
run_constrained:
- libcufile-static >={{ version }}
test:
commands:
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
- test -L $PREFIX/lib/libcufile.so
- test -f $PREFIX/targets/{{ target_name }}/include/cufile.h
- test -L $PREFIX/targets/{{ target_name }}/lib/libcufile.so
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: libcufile-static
files:
- targets/{{ target_name }}/lib/libcufile*_static.a
requirements:
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
test:
commands:
- test -f $PREFIX/lib/libcufile_static.a
- test -f $PREFIX/lib/libcufile_rdma_static.a
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: gds-tools
files:
- gds/samples
- gds/tools
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
host:
- cuda-version {{ cuda_version }}
- libcufile-dev {{ version }}
- libnuma # [linux]
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
- {{ pin_subpackage("libcufile", max_pin="x") }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
test:
commands:
- test -d $PREFIX/gds/samples
- test -d $PREFIX/gds/tools
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

extra:
recipe-maintainers:
- adibbley