Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARM OSX Migrator #14

Closed
wants to merge 30 commits into from
Closed

ARM OSX Migrator #14

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
6815865
migration: OSXArm
regro-cf-autotick-bot Jun 12, 2023
1d96bdf
MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…
regro-cf-autotick-bot Jun 12, 2023
da06a9c
MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…
Jun 18, 2023
1e95e15
test adding more flags to cmake
hadim Jun 18, 2023
c3580e7
WIP
hadim Jun 18, 2023
c86fa6a
WIP
hadim Jun 18, 2023
4419186
WIP
hadim Jun 18, 2023
8d32484
WIP
hadim Jun 18, 2023
1597eb0
WIP
hadim Jun 18, 2023
5a4015d
WIP
hadim Jun 18, 2023
08636d8
WIP
hadim Jun 18, 2023
f16dc0f
Update meta.yaml
hadim Jun 19, 2023
30a4067
Update build.sh
hadim Jun 19, 2023
9fdd392
WIP
hadim Jun 19, 2023
89f89b0
WIP
hadim Jun 19, 2023
33b7693
WIP
hadim Jun 19, 2023
dbc6186
WIP
hadim Jun 19, 2023
fa45a45
WIP
hadim Jun 19, 2023
afb834a
WIP
hadim Jun 19, 2023
a525133
WIP
hadim Jun 19, 2023
a7765f4
WIP
hadim Jun 19, 2023
7bdc4e7
WIP
hadim Jun 19, 2023
28de961
WIP
hadim Jun 19, 2023
5a8870e
WIP
hadim Jun 19, 2023
178d1b2
WIP
hadim Jun 19, 2023
7cedc43
WIP
hadim Jun 19, 2023
c719e7e
WIP
hadim Jun 19, 2023
28a2657
MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…
hadim Jun 19, 2023
562d253
WIP
hadim Jun 19, 2023
f4f615f
MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…
hadim Jun 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .azure-pipelines/azure-pipelines-osx.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions ...piler_version10cuda_compiler_version11.2cxx_compiler_version10python3.10.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
1 change: 1 addition & 0 deletions ...mpiler_version10cuda_compiler_version11.2cxx_compiler_version10python3.8.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
1 change: 1 addition & 0 deletions ...mpiler_version10cuda_compiler_version11.2cxx_compiler_version10python3.9.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
3 changes: 2 additions & 1 deletion ...piler_version12cuda_compiler_versionNonecxx_compiler_version12python3.10.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ channel_sources:
channel_targets:
- conda-forge main
cuda_compiler:
- nvcc
- None
cuda_compiler_version:
- None
cxx_compiler:
Expand All @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
3 changes: 2 additions & 1 deletion ...mpiler_version12cuda_compiler_versionNonecxx_compiler_version12python3.8.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ channel_sources:
channel_targets:
- conda-forge main
cuda_compiler:
- nvcc
- None
cuda_compiler_version:
- None
cxx_compiler:
Expand All @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
3 changes: 2 additions & 1 deletion ...mpiler_version12cuda_compiler_versionNonecxx_compiler_version12python3.9.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ channel_sources:
channel_targets:
- conda-forge main
cuda_compiler:
- nvcc
- None
cuda_compiler_version:
- None
cxx_compiler:
Expand All @@ -31,6 +31,7 @@ target_platform:
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- cuda_compiler
- cuda_compiler_version
- cdt_name
- docker_image
33 changes: 33 additions & 0 deletions .ci_support/osx_arm64_python3.10.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
MACOSX_DEPLOYMENT_TARGET:
- '11.0'
c_compiler:
- clang
c_compiler_version:
- '15'
channel_sources:
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler_version:
- None
cxx_compiler:
- clangxx
cxx_compiler_version:
- '15'
llvm_openmp:
- '15'
macos_machine:
- arm64-apple-darwin20.0.0
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.10.* *_cpython
pytorch:
- '2.0'
target_platform:
- osx-arm64
zip_keys:
- - c_compiler_version
- cxx_compiler_version
33 changes: 33 additions & 0 deletions .ci_support/osx_arm64_python3.8.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
MACOSX_DEPLOYMENT_TARGET:
- '11.0'
c_compiler:
- clang
c_compiler_version:
- '15'
channel_sources:
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler_version:
- None
cxx_compiler:
- clangxx
cxx_compiler_version:
- '15'
llvm_openmp:
- '15'
macos_machine:
- arm64-apple-darwin20.0.0
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.8.* *_cpython
pytorch:
- '2.0'
target_platform:
- osx-arm64
zip_keys:
- - c_compiler_version
- cxx_compiler_version
33 changes: 33 additions & 0 deletions .ci_support/osx_arm64_python3.9.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
MACOSX_DEPLOYMENT_TARGET:
- '11.0'
c_compiler:
- clang
c_compiler_version:
- '15'
channel_sources:
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler_version:
- None
cxx_compiler:
- clangxx
cxx_compiler_version:
- '15'
llvm_openmp:
- '15'
macos_machine:
- arm64-apple-darwin20.0.0
pin_run_as_build:
python:
min_pin: x.x
max_pin: x.x
python:
- 3.9.* *_cpython
pytorch:
- '2.0'
target_platform:
- osx-arm64
zip_keys:
- - c_compiler_version
- cxx_compiler_version
3 changes: 3 additions & 0 deletions .scripts/build_steps.sh
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions .scripts/run_osx_build.sh
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 21 additions & 0 deletions README.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions conda-forge.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
build_platform:
osx_arm64: osx_64
conda_build:
pkg_format: '2'
conda_forge_output_validation: true
github:
branch_name: main
tooling_branch_name: main
conda_forge_output_validation: true
conda_build:
pkg_format: '2'
test: native_and_emulated
5 changes: 5 additions & 0 deletions recipe/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,10 @@ export Torch_DIR=$(python -c 'import torch; print(torch.utils.cmake_prefix_path)
export FORCE_NINJA=1
export EXTERNAL_PHMAP_INCLUDE_DIR="${BUILD_PREFIX}/include/"
export EXTERNAL_CUTLASS_INCLUDE_DIR="${BUILD_PREFIX}/include/"
export PYG_CMAKE_ARGS="${CMAKE_ARGS} -DPython3_EXECUTABLE=${PYTHON}"

if [ "$target_platform" = "osx-arm64" ]; then
export PYG_CMAKE_ARGS="${PYG_CMAKE_ARGS} -DCMAKE_OSX_ARCHITECTURES=arm64"
fi

python -m pip install . -vvv
14 changes: 14 additions & 0 deletions recipe/fix-cmake.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
diff --git a/setup.py b/setup.py
index b9158f7..1a85091 100644
--- a/setup.py
+++ b/setup.py
@@ -77,6 +77,9 @@ class CMakeBuild(build_ext):
warnings.warn("Building times of 'pyg-lib' can be heavily improved"
" by installing 'ninja': `pip install ninja`")

+ if os.getenv("PYG_CMAKE_ARGS") is not None:
+ cmake_args += os.getenv("PYG_CMAKE_ARGS").split(" ")
+
build_args = []

subprocess.check_call(['cmake', ext.sourcedir] + cmake_args,
9 changes: 7 additions & 2 deletions recipe/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,20 @@ source:
url: https://github.com/pyg-team/pyg-lib/archive/refs/tags/{{ version }}.tar.gz
sha256: 3c486a72a421b4769cd1ca114522eedf25d4115ca529091db06c10264a406ac4

patches:
- fix-cmake.patch

build:
number: 2
number: 10
string: cuda{{ cuda_compiler_version | replace('.', '') }}py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }} # [cuda_compiler_version != "None"]
string: cpu_py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }} # [cuda_compiler_version == "None"]
# Only build for CUDA >=11.2
skip: true # [win or cuda_compiler_version in ("10.2", "11.0", "11.1")]

requirements:
build:
- python # [build_platform != target_platform]
- cross-python_{{ target_platform }} # [build_platform != target_platform]
hadim marked this conversation as resolved.
Show resolved Hide resolved
- {{ compiler('c') }}
- {{ compiler('cxx') }}
- {{ compiler('cuda') }} # [cuda_compiler_version not in (undefined, 'None')]
Expand All @@ -28,7 +33,7 @@ requirements:
- pytorch # [build_platform != target_platform]
- pytorch =*={{ torch_proc_type }}* # [build_platform != target_platform]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd try removing these two, because you're picking up pytorch from the build environment rather than host.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Trying that now. But then I am confused, does that mean you don't need arm64 pytorch at all for the cross compilation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example of a pytorch compiled package that has an osx arm64 build?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a bunch of them if you go to the conda-forge GH org and search for repos starting with torch or pytorch (or you can look which feedstocks get touched by the pytorch migration on conda-forge.org/status)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked torchani for example and indeed they dont have pytorch in build. But If I remove it here, then all the build fails.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've often seen it in build as well (presumably for anything that actually needs to execute some code from pytorch during installation), but for linking we need the one from host.

The right way would probably be to re-add the build dependency but ensure that whatever feeds the linkage mechanism looks in host first (might need a small patch)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build dependency is already here.

The current hack works, and it's already used by other packages. I understand it's far from ideal, but I don't really see another way at the moment.

- cmake
- ninja
- ninja # NOTE(hadim): here ninja is used for linux and osx
- llvm-openmp # [osx]
- libgomp # [linux]
- parallel-hashmap
Expand Down