Add macOS support #602

JayFoxRox · 2023-08-27T21:08:03Z

Introduction

This adds macOS support in chipStar and its underlying infrastructure (LLVM, HIP, HIPCC).
To make this even more interesting, this is on M1 (Apple Silicon), e.g. ARM.

I assume this will take a while to get merged.
For now, I hope that others can help to fix Windows / Linux things that I break in this macOS port.

OpenCL / SPIR-V on macOS

Obviously there isn't a metal backend currently (and I don't plan to write one), although I theorize it might be much easier to write once this has been merged.

This probably won't work with the Apple OpenCL implementation due to the lack of SPIR-V support.
Instead, I added support for the OpenCL ICD loader and I use a hacked version of pocl from brew for which I enabled SPIR-V support.
I don't care about performance right now either - all I want is to run some CUDA and HIP projects on macOS.. so chipStar it is!

Installation

Once everything is merged, I plan to create a Formula for brew (in homebrew-core), so this can be installed more easily (brew install chipstar).
However, I expect this can only happen once more of the chipStar changes in HIP, HIPCC and LLVM have been integrated in their respective upstreams (so that we don't need the forks in brew).
We could probably have a CHIP-SPV/homebrew-chipStar or similar though, to distribute this to macOS users.

For now, if you want to test this (on macOS):

Cloning

Follow steps in README for structure, but clone this branch.
For each submodule, check the TODO section below and checkout the corresponding branches.

Also see the note about pocl in the next section.

Install dependencies

# Install OpenCL base
brew install opencl-headers
brew install opencl-icd-loader

# Install pocl with SPIR-V support
brew install spirv-llvm-translator
brew install pocl # Needs SPIR-V support, see below

Set up some config:

export J=8
export CHIPSTAR_LLVM_INSTALL=/tmp/chipstar-llvm-install/
export CHIPSTAR_INSTALL=/tmp/chipstar-install/
export SDK_PATH=$(xcrun --show-sdk-path)
export BUILD_TYPE=Debug
export BUILD_NAME=build-debug

Build LLVM:

cmake -S llvm -B $BUILD_NAME \
  -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
  -DLLVM_ENABLE_PROJECTS="clang;openmp" \
  -DLLVM_TARGETS_TO_BUILD="host" \
  -DDEFAULT_SYSROOT=$SDK_PATH \
  -DCMAKE_INSTALL_PREFIX=$CHIPSTAR_LLVM_INSTALL
make -C $BUILD_NAME -j $J all install

Build chipStar:

mkdir $BUILD_NAME && cd $BUILD_NAME
cmake .. \
    -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
    -DLLVM_CONFIG_BIN=$CHIPSTAR_LLVM_INSTALL/bin/llvm-config \
    -DCMAKE_INSTALL_PREFIX=$CHIPSTAR_INSTALL \
    "-DCMAKE_PREFIX_PATH=$HOMEBREW_PREFIX/opt/opencl-icd-loader/share/cmake;$HOMEBREW_PREFIX/opt/opencl-headers/share/cmake/"
make all build_tests install -j $J

Testing:

These are mostly notes to myself, but this is how to test:

CHIP_LOGLEVEL=trace $CHIPSTAR_INSTALL/bin/chip_spv_samples/hipmath

make CHIP && bin/hipcc.bin ../samples/hipmath/hipmath.cc && POCL_DEBUG=all POCL_LEAVE_KERNEL_COMPILER_TEMP_FILES=1 CHIP_DUMP_SPIRV=1 ./a.out

Status

Compiles
Tests run, (but results not verified yet)
Samples run, (but results not verified yet)
Never used in a real application

TODO

JayFoxRox · 2023-08-27T21:16:24Z

include/CL/opencl.hpp

+#define CL_DEPRECATED(...)
+#define GCL_API_SUFFIX__VERSION_1_1
+//#include <OpenCL/opencl.h>
+#include <CL/opencl.h>


Can probably be changed back; I believe ICD Loader also overloads <OpenCL/opencl.h>

JayFoxRox · 2023-08-27T21:16:36Z

CMakeLists.txt

  list(APPEND CHIP_SPV_DEFINITIONS HAVE_OPENCL)
-  list(PREPEND CHIP_INTERFACE_LIBS ${OpenCL_LIBRARY})
+  list(PREPEND CHIP_INTERFACE_LIBS 
+  #OpenCL::Headers


JayFoxRox · 2023-08-27T21:17:31Z

CMakeLists.txt

@@ -114,10 +140,10 @@ if(NOT DEFINED LevelZero_LIBRARY)
  endif()
 endif()

-message(STATUS "OpenCL_LIBRARY: ${OpenCL_LIBRARY}")
+message(STATUS "OpenCL_FOUND: ${OpenCL_FOUND}")


This is obviously ugly; I'll have to figure out how to show the path with import targets.
Maybe it's also enough to report YES / NO for each of these.

JayFoxRox · 2023-08-27T21:17:41Z

CMakeLists.txt

+  find_package(OpenCLHeaders)
+  find_package(OpenCLICDLoader)
+
+  #add_library(OpenCL ALIAS OpenCLICDLoader)


JayFoxRox · 2023-08-27T21:17:50Z

CMakeLists.txt

+  #add_library(OpenCL ALIAS OpenCLICDLoader)
+  message("OpenCL_FOUND: ${OpenCL_FOUND}")
+  set(OpenCL_FOUND ${OpenCLICDLoader_FOUND})
+  message("OpenCL_FOUND: ${OpenCL_FOUND}")


Debug leftover

JayFoxRox · 2023-08-27T21:18:12Z

CMakeLists.txt

+  set(ARCH "x64")
+else()
+  set(ARCH "unknown")
+endif()


This entire block is super ugly. I'm not sure how to do this best

JayFoxRox · 2023-08-27T21:18:50Z

CMakeLists.txt

+  add_compile_options(-mf16c)
+elseif("${CMAKE_C_COMPILER_ARCHITECTURE_ID}" STREQUAL "x64")
+  add_compile_options(-mf16c)
+endif()


How important is -mf16c for x86? Could it be removed? It's in a "temporary" block, too.

I recall, the -mf16c is required for host side native half type and operations (hip_fp16.h).

Correction: the -mf16c is used for working around an undefined reference issue. However, the same issue is seen here too despite the workaround so I’m not sure if the option is actually effective.

JayFoxRox · 2023-08-27T21:19:26Z

include/hip/devicelib/macros.hh

@@ -52,4 +52,7 @@
 typedef _Float16 api_half;
 typedef _Float16 api_half2 __attribute__((ext_vector_type(2)));

+typedef unsigned int uint;
+typedef unsigned long ulong;


Why is this necessary on macOS? Does it still work for other platforms or will it report duplicate identifier?

It might cause troubles. On my system (Ubuntu 22.04) <stdlib.h> includes <sys/types.h> which defines typedefs for u{short,int,long}.

/usr/include/x86_64-linux-gnu/sys/types.h:

... #ifdef __USE_MISC /* Old compatibility names for C types. */ typedef unsigned long int ulong; typedef unsigned short int ushort; typedef unsigned int uint; #endif ...

Actually, it's not a problem as long as the duplicate typedefs aliases the same type.

JayFoxRox · 2023-08-27T21:22:11Z

llvm_passes/CMakeLists.txt

+  target_link_libraries(LLVMHipDynMem "-undefined dynamic_lookup")
+  target_link_libraries(LLVMHipStripUsedIntrinsics "-undefined dynamic_lookup")
+  target_link_libraries(LLVMHipDefrost "-undefined dynamic_lookup")
+  target_link_libraries(LLVMHipPasses "-undefined dynamic_lookup")


These should potentially be set_target_properties(LLVMHipDynMem PROPERTIES LINK_FLAGS "-undefined dynamic_lookup) etc.; I'll have to read up on it.

I'm not sure why this is necessary in the first place (I think it should be the default for MODULE libs)

Works for now.

JayFoxRox · 2023-08-27T21:22:44Z

llvm_passes/HipAbort.cpp

    InverseCallGraphNode *N = popAny(WorkList);
+    if (N == nullptr) {
+      printf("N was nullptr! skipping\n");


This gets triggered in the abort sample. Why?
Is this expected or a macOS issue?

JayFoxRox · 2023-08-27T21:24:06Z

samples/CMakeLists.txt

@@ -129,8 +129,14 @@ set(SAMPLES
    ccompat
    hipComplex
    hipHostMallocSample
+)
+# We skip this test on macOS, because macho does not support EXCLUDE section flags (used by clang-offload-bundler)


I plan to fix clang-offload-bundler eventually, but I'm new to most of LLVM, mach-o, HIP, OpenCL and chipStar.
So I'll have to understand why this is necessary in the first place / what exactly it does.

JayFoxRox · 2023-08-27T21:24:43Z

src/CHIPBackend.cc

@@ -1257,7 +1257,12 @@ void chipstar::Backend::waitForThreadExit() {
   *
   * So we just wait for 0.5 seconds before starting to check for thread exit.
   */
+#if defined(__APPLE__) || defined(__MACOSX)
+  sched_yield();


JayFoxRox · 2023-08-27T21:26:42Z

src/CHIPDriver.cc

@@ -77,7 +77,7 @@ void CHIPReadEnvVarsCallOnce() {

  CHIPDeviceTypeStr = readEnvVar("CHIP_DEVICE_TYPE");
  if (CHIPDeviceTypeStr.size() == 0)
-    CHIPDeviceTypeStr = "gpu";
+    CHIPDeviceTypeStr = "default";


After spending half an hour until realizing pocl was missing SPIR-V, I've spent another half an hour trying to figure out why chipStar still didn't work, until I realized it was filtering out my CPU device.
I've tried to find a good solution but the CL spec was lacking. Would "default" type still work for most people? (or will they be surprised if their OpenCL platform suddenly kicks them from CPU to GPU after a chipStar update?)

JayFoxRox · 2023-08-27T21:27:52Z

tests/runtime/TestStlFunctions.hip

  launchUnaryFn<float>([] __device__(auto x) { return std::abs(x); });
+#endif


I'm getting warnings about this being ambiguous, I can add a log with these errors later.
Why isn't this an issue for other platforms?

I can add a log with these errors later.

Please, send us the log.

pvelesko · 2023-08-28T09:35:49Z

cool! I will try this on my m1 this week

pjaaskel · 2023-08-28T13:48:55Z

Nice! Thanks for working on this. For the "Metal backend" -- you might do more good contributing that backend to PoCL instead. You'd get at least SYCL support (via DPC++ or OpenSYCL) too that way as a side effect.

Kerilk · 2023-08-28T15:17:25Z

@JayFoxRox Thank you so much for this contribution! We discussed it extensively this morning,and we would very much like to merge it when it is ready. I order to do so we will need CI support.
Our plan here is to start leveraging GitHub action to test CPU pocl for Ubuntu, and then we should be able to extend it for MacOS. This should prevent things from bit-rotting without adding additional burden on the few GPU machines we have available.

JayFoxRox · 2023-08-28T22:48:42Z

For the "Metal backend" -- you might do more good contributing that backend to PoCL instead. You'd get at least SYCL support (via DPC++ or OpenSYCL) too that way as a side effect.

Yes, pocl integration was also what I was thinking.
I also toyed around with spirv-cross and spirv-tools earlier today and noticed that they recently added support for SPIR-V compute to metal compute. This means that even much of the SPIR-V stuff is already done!

However, for chipStar, it might be interesting to add a Vulkan backend, because that would probably work with MoltenVK, so you'd get the rest of the metal API "for-free" + you'd also get support for a bunch of Vulkan devices which lack OpenCL (or proper drivers).
Even then you might still be able to do SYCL (unless I'm missing something - didn't read much into SYCL yet):

SYCL application > OpenSYCL > OpenSYCL-HIP-Backend > chipStar > chipStar-Vulkan-Backend > Vulkan intermediate > MoltenVK > Metal

I probably won't find time to ever work on that. I'm fine with having CPU-only (for now). For me, it's mostly about getting a bit into HIP/CUDA + I want to run some projects which are CUDA only.

cool! I will try this on my m1 this week

Note that it doesn't really work yet, but patches are welcome.
I still have to figure out how to resolve (or even reproduce with OpenCL) the mangling issues in pocl/pocl#1288.

Our plan here is to start leveraging GitHub action to test CPU pocl for Ubuntu, and then we should be able to extend it for MacOS.

Sounds good.

pjaaskel · 2023-08-29T11:09:24Z

However, for chipStar, it might be interesting to add a Vulkan backend, because that would probably work with MoltenVK, so you'd get the rest of the metal API "for-free" + you'd also get support for a bunch of Vulkan devices which lack OpenCL (or proper drivers). Even then you might still be able to do SYCL (unless I'm missing something - didn't read much into SYCL yet):

I forgot that there's already also a starting point PoCL-Vulkan backend (quite early and primitive) that could be improved for MacOS needs if it supports Vulkan.

* SYCL application > OpenSYCL > OpenSYCL-HIP-Backend > chipStar > chipStar-Vulkan-Backend > Vulkan intermediate > MoltenVK > Metal

Why not... To me HIP is a higher level than OpenCL though so I'd simplify this to SYCL -> DPC++/OpenSYCL -> OpenCL -> X instead of hopping through HIP for SYCL support.

isuruf · 2023-09-07T02:18:47Z

I've tried the pocl-vulkan backend with macos-m1, but couldn't get it to work. I also tried writing a Metal backend for PoCL, and made some progress, but the Metal documentation was very hard to digest. (For eg: how to use events)

pjaaskel · 2023-09-07T14:14:37Z

@isuruf is Metal backend starting point usable at all? I suggest to contribute it to the main if it runs any examples so it's easier for other volunteers to contribute to make it better.

JayFoxRox · 2023-09-07T14:58:50Z

I've tried the pocl-vulkan backend with macos-m1, but couldn't get it to work. I also tried writing a Metal backend for PoCL

So far, I've only tried the CPU backend, but I ran into a bunch of issues with PoCL (partially issues in the brew package, but also in PoCL itself) which is also why there hasn't been much progress here. Until there's a solution for pocl/pocl#1288 I can't really continue on the chipStar side either - I can't really know if the issues I'm seeing are from PoCL or from chipStar.
However, I don't have time to also work on on PoCL and am not sure how to continue.

isuruf · 2023-09-07T15:53:50Z

@isuruf is Metal backend starting point usable at all?

No, it's not.

JayFoxRox added 19 commits August 27, 2023 17:30

Use CMake to install CHIP target instead of installing files

5b49a93

Use PASS_NAME in hip-lower-switch

36384f1

Set default CHIP_DEVICE_TYPE to 'default'

cb7105e

Fix LogLevel variable name in commented out code

0de3e36

Upgrade spdlog v1.12.0

7c717e9

Use ICD loader by default

ebdb4d9

Avoid x86 flags on ARM

a932459

Fix bug with printing pthread_t on macOS

8d7937e

Fix bug with missing uint and ulong types

031cef5

Force dynamic lookup on macOS for undefined references

13d0052

Use sched_yield instead of pthread_yield on macOS

e4db075

Add macOS support to samples

4c3a586

Disable hipDeviceLink on macOS (no offload-bundler / RDC)

ddc56b0

HACK to disable workaround for issue-102

648febc

HACK Debug bug in abort pass

d674499

disablebrokenabort

159aa43

disableparserendoffile

f0b9328

Fix ccompat which depends on generated headers

c6c4af8

REVIEW Disable tests with ambiguous math functions

f6a556d

This was referenced Aug 27, 2023

LLVM-16: Add macOS support CHIP-SPV/llvm-project#1

Draft

Add macOS support CHIP-SPV/HIP#5

Draft

Add macOS support CHIP-SPV/HIPCC#5

Draft

JayFoxRox commented Aug 27, 2023

View reviewed changes

CMakeLists.txt

find_package(OpenCLHeaders)

find_package(OpenCLICDLoader)

#add_library(OpenCL ALIAS OpenCLICDLoader)

Copy link

Author

JayFoxRox Aug 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead code

JayFoxRox commented Aug 27, 2023

View reviewed changes

Kerilk mentioned this pull request Sep 18, 2023

Support Apple's OpenCL.framework? OCL-dev/ocl-icd#31

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add macOS support #602

Add macOS support #602

JayFoxRox commented Aug 27, 2023 •

edited

Loading

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

linehill Aug 28, 2023

linehill Aug 29, 2023

JayFoxRox Aug 27, 2023

linehill Aug 29, 2023

linehill Aug 29, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023

JayFoxRox Aug 27, 2023 •

edited

Loading

JayFoxRox Aug 27, 2023

linehill Aug 29, 2023

pvelesko commented Aug 28, 2023

pjaaskel commented Aug 28, 2023

Kerilk commented Aug 28, 2023

JayFoxRox commented Aug 28, 2023 •

edited

Loading

pjaaskel commented Aug 29, 2023

isuruf commented Sep 7, 2023

pjaaskel commented Sep 7, 2023

JayFoxRox commented Sep 7, 2023

isuruf commented Sep 7, 2023

		launchUnaryFn<float>([] __device__(auto x) { return std::abs(x); });
		#endif

Add macOS support #602

Are you sure you want to change the base?

Add macOS support #602

Conversation

JayFoxRox commented Aug 27, 2023 • edited Loading

Introduction

OpenCL / SPIR-V on macOS

Installation

Status

TODO

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JayFoxRox Aug 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvelesko commented Aug 28, 2023

pjaaskel commented Aug 28, 2023

Kerilk commented Aug 28, 2023

JayFoxRox commented Aug 28, 2023 • edited Loading

pjaaskel commented Aug 29, 2023

isuruf commented Sep 7, 2023

pjaaskel commented Sep 7, 2023

JayFoxRox commented Sep 7, 2023

isuruf commented Sep 7, 2023

JayFoxRox commented Aug 27, 2023 •

edited

Loading

JayFoxRox Aug 27, 2023 •

edited

Loading

JayFoxRox commented Aug 28, 2023 •

edited

Loading